Grade Check CLI
A technical reference for instructors and course staff who use the Grade Check command group in the Nuvolos CLI.
Overview
Grade Check is an instructor tool built into the Nuvolos CLI. It replaces the manual grading loop - starting each Student Instance, running your test script, saving the output, and stopping the Application again - with a single command.
What it’s good for
Use case
Supported
Auto-checking a homework script or notebook test command
Yes
Running the same shell command across many Student Instances
Yes
Collecting submissions first, then validating separately
Yes
Dry-run planning without starting any Applications
Yes
Parallel checks across several students at once
Yes
How it works
Grading 40 homework submissions by hand typically means opening each student’s workspace, starting their Application, running your test script, saving the output, stopping the Application so Credits aren’t wasted, and returning feedback where the student can see it.
Grade Check automates that entire checklist. At a high level, a run has three stages:
Collect submissions - Copy student hand-ins into a working folder and build a manifest (a roster of every submission).
Process each student - For every Student Instance: start the Application, run your validation command, pull the logs, then stop the Application.
Publish results - Write output.log and grade_meta.json into both the handin area (instructor-visible) and the handback area (student-visible).
Regardless of role, you’ll always need:
Access to the teaching Space’s Master Instance.
An instructor-level API key (or you can run the command directly on Nuvolos, where the context is already set for you).
A clear validation command that each student’s Application can run - for example, python
check_assignment.py.
The per-student lifecycle in detail
For each student, Grade Check performs the following steps in order:
Preflight - confirm the Application exists on that Instance.
Start the Application.
Wait until the Application status is RUNNING.
Execute your validation command (working directory:
/files).Wait until the command finishes, or the timeout is reached.
Pull logs to the instructor machine.
Stop the Application - this is always attempted, even if the validation step failed.
Publish artifacts under handin and handback.
Key concepts
Teaching Space and Master Instance
Grade Check must be run from the Master Instance of a Course.
Course - a Teaching Space that contains Student Instances.
Master Instance - the instructor’s “home base” Instance (slug master).
Student Instance - one or more per student.
If you run the command from a Student Instance or from a research Space, it will refuse to start.
Assignment hand-in and handback
Nuvolos stores assignment review files under two paths:
Path
Who sees it
Purpose
/files/assignments-review/handin/...
Instructors
Original submissions and grading copies
/files/assignments-review/handback/...
Students (read-only)
Feedback returned to students
Grade Check writes its own artifacts into a dedicated subfolder so they never collide with a student’s own files (for example, a student’s own output.log):
.../<submission>/_grading/output.log
.../<submission>/_grading/grade_meta.json
Manifest
After collect, a file named nvcollect_manifest.json lists every submission, including:
Where it came from (src)
Where it was copied to (target)
Which Student Instance it belongs to
Grade Check uses this file as the roster of students to test.
Validation command
This is your shell command - for example, python assignments/hw1/check.py. A few rules of thumb apply:
Exit code 0 means success (the student’s submission is marked “ok”).
Any non-zero exit code means failure.
stdout and stderr are both captured into output.log.
The following placeholders are available: {instance_slug}, {instance}, and {target}.
Staging vs. durable storage
Kind
Default location
Kept permanently?
Staging (scratch space while running)
/files/.nuvolos_grade/<run_id>/...
No - working area only
Durable (final artifacts)
handin and handback trees
Yes — this is what you and your students us-
In most cases, you won’t need to set a results directory yourself.
Prerequisites
People and roles
You’re an Instructor or TA with access to the Course.
You can open the Master Instance.
Platform
A Nuvolos Course Space.
Student Instances visible to your account.
The same Application slug present on every Student Instance (for example, jupyter, rstudio, or a custom course image).
Software
Component
Required?
Notes
nuvolos-cli
Yes
Provides the nuvolos grade command group.
nuvolos_collect (nv-collect)
Yes, for collect
Installed separately — see Installation.
API key
Yes, off-platform
Often already available automatically when running on Nuvolos.
NV_CONTEXT
Yes, for grade check
Set automatically inside Nuvolos Applications.
Test your validation script first
Before running Grade Check at scale:
Open one Student Instance manually.
Start the Application you plan to use.
Run the exact command you intend to pass to --command.
Confirm the exit code and output make sense.
Installation
Install the CLI
pip install nuvolos-cli
Full documentation is available under Installing the CLI or at Nuvolos CLI.
Install collect support
grade collect, and the collect step inside grade check, both depend on nuvolos_collect. This package isn’t bundled as a pip extra, due to a dependency conflict between package versions on PyPI, so it must be installed separately:
pip install git+https://github.com/alphacruncher/nv-collect.git
If this package is missing, the CLI will tell you to install it.
Configure API access
If you aren’t already running inside Nuvolos, configure your API key with either of the following:
For additional debug output, you can optionally set:
export NUVOLOS_CLI_DEBUG=true
Before you run anything
Checklist
You’re in the Master Instance of the Course.
You know your Organization slug (-o) and Space slug (-s).
You know the Application slug used on Student Instances (-a).
You know the assignment name and assignment folder used in hand-ins.
Your validation command has already been tested on a sample student.
nuvolos_collectis installed, if you plan to collect submissions.Your API key works (nuvolos instances list -o ... -s ... succeeds).
Discover your slugs
nuvolos orgs list
nuvolos spaces list -o <org>
nuvolos instances list -o <org> -s <space>
nuvolos apps list -o <org> -s <space> -i master
# And on a Student Instance:
nuvolos apps list -o <org> -s <space> -i <student_instance>
Assignment name vs. assignment folder
These two flags map to nvcollect conventions under the hand-in tree:
Flag
Meaning
Example
--assignment-name
Bundle/assignment name under handin
homework1
--assignment-folder
Leaf folder name inside each submission
hw1 or submission
If you already use nvcollect or the assignments-review UI, reuse the same names you use there.
Quick start
Step 1: First run - plan only, no Applications started
Step 2: Real run, one student
Step 3: Whole class
What success looks like
Near the end of the run, you’ll see a summary similar to this:
The run ID is a UTC timestamp, such as 20260910T143015Z. For each student, the CLI also prints the status, exit code, handin/handback paths, and — often — the full output.log contents.
Command reference
The grade feature is a command group with three subcommands:
nuvolos grade ├── collect Collect submissions only ├── resolve-manifest Preview who would be graded from a manifest └── check Full pipeline: collect → test → publish
Nuvolos grade check (main command)
Purpose: Collect submissions (optionally), run your validation command on each Student Instance, and publish the results.
nuvolos grade check [OPTIONS]
Required flags
Option
Short
Description
--org
-o
Organization slug
--space
-s
Space slug
--app-slug
-a
Application slug to start on each Student Instance
--command
-c
Shell command to run inside each student’s Application
Also required, unless --skip-collect is set:
Option
Description
--assignment-name
Assignment/bundle name under handin
--assignment-folder
Leaf folder inside each submission
With --skip-collect, provide --manifest (or the deprecated --target-folder) pointing at an existing collect tree or manifest.
Common optional flags
Option
Default
Description
--instance / -i
all
Limit to a single Student Instance slug
--limit N
none
Process at most N students
--all
off
Grade every resolved student (clears --limit)
--parallel N
1
Number of concurrent student checks
--continue-on-error
off
Keep going after the first failure
--skip-missing-instances
off
Skip students not visible in the Space
--dry-run
off
Plan only — no start, execute, or stop
--skip-collect
off
Reuse an existing manifest / prior collect
--manifest / -m
—
Manifest path, when skipping collect
--pull-logs / --no-pull-logs
pull on
Pull and publish output.log
--instructor-instance
master (or context)
Where logs are distributed
Advanced flags
Option
Notes
--results-dir / -r
Overrides the staging root (default /files/.nuvolos_grade). Durable output still goes to handin/handback.
--target-folder
Deprecated for normal checks; with --skip-collect, can point at an existing collect tree.
--test-command
Deprecated alias for --command.
Exit behavior: If any student fails and the run isn’t a dry run, the CLI exits with an error after printing the summary. Individual validation exit codes are stored in grade_meta.json and in the summary.
Nuvolos grade collect
Purpose: Gather submissions only - no start or execute step.
Option
Required
Description
--assignment-name
Yes
Bundle name under handin
--assignment-folder
Yes
Leaf folder name
--target-folder
Yes
Destination for the collected trees and nvcollect_manifest.json
--org / -o
Recommended
Enables renaming folders to student email
--space / -s
Recommended
Same as above
Without --org/--space, folders may stay named by Instance slug instead of student email.
Internally, this calls nuvolos_collect.collect, then optionally renames folders to student email and rewrites the manifest.
nuvolos grade resolve-manifest
Purpose: Preview the roster — map manifest rows to Instances and check visibility before a large run.
Option
Description
--manifest / -m
Manifest file or parent directory
--org / -o, --space / -s
Required
--instance / -i
Optional filter
-f (table or json)
Output format (default table)
Example table columns: instance_slug | email | name | in_space | target
Common workflows
Workflow 1 — Happy path: collect and grade in one go
Use this when hand-ins are already in the review area and you want a single command.
Workflow 2 — Collect once, grade many times
Useful while you’re still tuning the validation command.
Workflow 3 — Grade a single student (debug)
Workflow 4 — Parallel batch with safe defaults
Important: Without --continue-on-error, parallel mode is forced back to 1 so the run can stop cleanly after the first failure.
Workflow 5 — Dry-run capacity planning
No Applications are started. This is useful for validating flags and environment gates before a real run.
Understanding results
Status values (per student)
Status
Meaning
executed
Lifecycle completed; validation exit code was 0.
failed
Application missing, start/execute error, non-zero exit code, publish failure, timeout, and similar.
skipped
Filtered out, missing Instance (when allowed), or left unprocessed after an earlier failure.
dry_run
Would have run, but --dry-run was set.
pending
Internal/incomplete — should be rare in a final summary.
How “ok” is counted
The summary’s ok count is approximately the number of students with status executed.
A non-zero validation exit code marks a student as failed, even if start and stop both worked correctly.
A publish error can flip an otherwise successful execution to failed.
What students see
After a successful publish and handback, students see files under handback, inside _grading/. These files are typically read-only and contain:
output.log— the command outputgrade_meta.json— a machine-readable summary
What instructors see
Live terminal progress and log excerpts.
A final GRADE CHECK SUMMARY.
The same artifacts under both handin (next to the submission) and handback.
Staging copies under
/files/.nuvolos_grade/<run_id>/<student-email>/while the run is active.
Typical grade_meta.json fields
Terminal summary structure
For each student, the CLI prints roughly the following:
Email / folder
Instance slug
Status (with exit code)
Error, if any
handin path
handback path (student-visible)
Published file paths
Local log path
Full or truncated output.log
A batch handback status appears once at the end of the run.
Options explained
--command / -c
Runs inside the student’s Application, with the working directory effectively under /files. For example:
Available placeholders (simple string replacement):
Placeholder
Expands to
{instance_slug}
Student Instance slug
{instance}
Same as {instance_slug}
{target}
Collect target path for that student (may be empty)
Tips for writing your validation command:
Prefer simple, non-interactive commands.
Print clear pass/fail messages to stdout.
Set exit codes deliberately (sys.exit(0) / sys.exit(1)).
Avoid relying on GUI applications.
--parallel
Defaults to 1 (sequential).
Higher values start multiple student lifecycles at once.
Requires --continue-on-error to actually run in parallel; otherwise the CLI forces sequential execution.
Start small — 2 to 4 concurrent checks — and monitor cluster capacity and course Credits as you scale up.
--continue-on-error
Without the flag
With the flag
The first hard failure stops the queue; remaining students are marked skipped.
The run keeps going through the full roster.
Parallel execution is forced to 1.
Parallel execution is allowed.
Recommendation: Always pass --continue-on-error for class-wide runs.
--limit vs. --all
--limit 5 processes the first 5 resolved students — useful for staged rollouts.
--all processes everyone and clears any limit.
--pull-logs / --no-pull-logs
Defaults to on. After execution, logs are distributed to the instructor Instance and published. Turn this off only for special-case debugging — you won’t get the usual handin/handback artifacts.
--instructor-instance
Defaults to the Instance from NV_CONTEXT, falling back to master. Logs are distributed to this Instance so the instructor can read them on the local /files tree.
Timeouts (environment variables)
Variable
Default
Meaning
GRADE_EXEC_TIMEOUT_SECS
600 (10 min)
Maximum wait for one student’s validation command
GRADE_STAGE_TIMEOUT_SECS
120
Maximum wait while staging logs before distribution
APP_START_TIMEOUT_SECS
600
Application start wait (shared across the CLI)
APP_TASK_TIMEOUT_SECS
600
Background task wait, e.g. distribute
export GRADE_EXEC_TIMEOUT_SECS=1200 # 20 minutes, for slow tests
File layout and artifacts
Staging layout (scratch)
/files/.nuvolos_grade/ └── <run_id>/ e.g. 20260910T143015Z ├── nvcollect_manifest.json (after collect) ├── alice@uni.edu/ │ ├── ...collected files... │ ├── output.log │ ├── .cmd_done │ └── .cmd_exit.0 └── bob@uni.edu/ └── ...
Folders are preferably named by student email, when org/Space resolution succeeds.
Durable layout (what matters long-term)
/files/assignments-review/ ├── handin/ │ └── <instance_slug>/<assignment>/<timestamp>_.../ │ ├── ...submission files... │ └── _grading/ │ ├── output.log │ └── grade_meta.json └── handback/ └── <instance_slug>/<assignment>/<timestamp>_.../ └── _grading/ ├── output.log ← student-visible, read-only └── grade_meta.json
Why the _grading/ subfolder?
So grading files never collide with a student’s own output.log, or other similarly named files, inside their submission.
Rules and limits
Hard gates (the command will refuse to run without these)
An API key must be configured.
NV_CONTEXT must match -o / -s.
The context Instance must be the Master Instance.
The Space type must be Course.
Operational safety
Concern
Behavior
Applications left running
Stop is always attempted in a finally block after start.
One student fails
Without --continue-on-error, remaining students are skipped.
Missing Application on an Instance
That student is marked failed; others may continue if allowed.
Missing Instance in the Space
Errors, or is skipped with --skip-missing-instances / --continue-on-error.
Credits and cost
Each Application start consumes Nuvolos Compute Units (NCUs) and draws down your Space’s Credits — prefer --limit first, then scale up.
Student-facing feedback
Handback files are made read-only on a best-effort basis.
What Grade Check does not guarantee yet
Automatic Credit protection or billing controls for student budgets — this is a platform-level concern and should be planned separately.
GPU-aware scheduling beyond what the Application start API already supports.
Keeping Applications running for external, interactive evaluation — that’s a different feature (see Related but separate: evaluation sessions).
Troubleshooting
“must run from the master instance”
You’re not on the Master Instance, or NV_CONTEXT points elsewhere. Open a terminal on the teaching Space’s Master Instance.
“only supported in teaching spaces”
Wrong Space type. Use the course teaching Space, not a research Space.
“nuvolos_collect is required”
Install: pip install git+https://github.com/alphacruncher/nv-collect.git
“App slug … not found on instance”
That Student Instance doesn’t have the Application installed. Distribute the course Application to all Instances, or filter with --instance, or fix the image.
“Timed out … waiting for command completion”
The validation command ran longer than GRADE_EXEC_TIMEOUT_SECS (10 minutes by default), or the done-marker never appeared.
Test the command manually on one student.
Increase GRADE_EXEC_TIMEOUT_SECS.
Make sure the command is non-interactive and eventually exits.
“Instance not found … (or API key lacks access)”
The manifest references a student your API key can’t see. Use --skip-missing-instances for partial runs, or fix access/roster.
Parallel execution didn’t parallelize
You forgot --continue-on-error. Without it, workers are forced to 1.
No logs, or an empty handback
Confirm --pull-logs is enabled (it’s on by default).
Check whether execution actually produced output.log.
Look at staging under /files/.nuvolos_grade/<run_id>/....
Confirm distribute permissions and the instructor Instance are correct.
Collect folder names are slugs, not emails
Pass -o and -s (or rely on NV_CONTEXT) so the email relabeling step can run.
Command failed, but I need the log
Even on failure, Grade Check tries to pull logs before stopping the Application. Check the summary’s log and handin paths, plus the terminal excerpt.
The whole CLI exits non-zero after partial success
This is by design: if failed > 0, grade check raises an error at the end so CI pipelines and scripts notice. Read the printed summary — individual students may still be fine.
Last updated
Was this helpful?