> For the complete documentation index, see [llms.txt](https://docs.nuvolos.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nuvolos.com/reference/nuvolos-cli-and-python-api/grade-check-cli.md). # Grade Check CLI A technical reference for instructors and course staff who use the Grade Check command group in the Nuvolos CLI. ## Overview Grade Check is an instructor tool built into the [Nuvolos CLI](https://nuvolos-cli.readthedocs.io/). It replaces the manual grading loop - starting each Student Instance, running your test script, saving the output, and stopping the Application again - with a single command. #### What it’s good for


Use case	Supported
Auto-checking a homework script or notebook test command	Yes
Running the same shell command across many Student Instances	Yes
Collecting submissions first, then validating separately	Yes
Dry-run planning without starting any Applications	Yes
Parallel checks across several students at once	Yes

#### How it works Grading 40 homework submissions by hand typically means opening each student’s workspace, starting their Application, running your test script, saving the output, stopping the Application so Credits aren’t wasted, and returning feedback where the student can see it. Grade Check automates that entire checklist. At a high level, a run has three stages: 1. **Collect submissions** - Copy student hand-ins into a working folder and build a manifest (a roster of every submission). 2. **Process each student -** For every Student Instance: start the Application, run your validation command, pull the logs, then stop the Application. 3. **Publish results** - Write output.log and grade\_meta.json into both the handin area (instructor-visible) and the handback area (student-visible). Regardless of role, you’ll always need: 1. Access to the teaching Space’s **Master Instance**. 2. An instructor-level API key (or you can run the command directly on Nuvolos, where the context is already set for you). 3. A clear **validation command** that each student’s Application can run - for example, python `check_assignment.py`. #### The per-student lifecycle in detail For each student, Grade Check performs the following steps in order: 1. Preflight - confirm the Application exists on that Instance. 2. Start the Application. 3. Wait until the Application status is RUNNING. 4. Execute your validation command (working directory: `/files`). 5. Wait until the command finishes, or the timeout is reached. 6. Pull logs to the instructor machine. 7. Stop the Application - this is always attempted, even if the validation step failed. 8. Publish artifacts under handin and handback. ## Key concepts #### Teaching Space and Master Instance Grade Check must be run from the **Master Instance** of a **Course**. * **Course** - a Teaching Space that contains Student Instances. * **Master Instance** - the instructor’s “home base” Instance (slug master). * **Student Instance** - one or more per student. If you run the command from a Student Instance or from a research Space, it will refuse to start. #### Assignment hand-in and handback Nuvolos stores assignment review files under two paths:


Path	Who sees it	Purpose
/files/assignments-review/handin/...	Instructors	Original submissions and grading copies
/files/assignments-review/handback/...	Students (read-only)	Feedback returned to students

Grade Check writes its own artifacts into a dedicated subfolder so they never collide with a student’s own files (for example, a student’s own output.log): `...//_grading/output.log`\ `...//_grading/grade_meta.json` #### Manifest After collect, a file named `nvcollect_manifest.json` lists every submission, including: * Where it came from (src) * Where it was copied to (target) * Which Student Instance it belongs to Grade Check uses this file as the roster of students to test. #### Validation command This is your shell command - for example, python `assignments/hw1/check.py`. A few rules of thumb apply: * Exit code 0 means success (the student’s submission is marked “ok”). * Any non-zero exit code means failure. * *stdout* and *stderr* are both captured into output.log. * The following placeholders are available: {instance\_slug}, {instance}, and {target}. #### Staging vs. durable storage


Kind	Default location	Kept permanently?
Staging (scratch space while running)	/files/.nuvolos_grade/<run_id>/...	No - working area only
Durable (final artifacts)	handin and handback trees	Yes — this is what you and your students us-

In most cases, you won’t need to set a results directory yourself. ## Prerequisites #### People and roles * You’re an Instructor or TA with access to the **Course**. * You can open the **Master Instance**. #### Platform * A Nuvolos Course Space. * Student Instances visible to your account. * The same Application slug present on every Student Instance (for example, jupyter, rstudio, or a custom course image). #### Software


Component	Required?	Notes
nuvolos-cli	Yes	Provides the nuvolos grade command group.
nuvolos_collect (nv-collect)	Yes, for collect	Installed separately — see Installation.
API key	Yes, off-platform	Often already available automatically when running on Nuvolos.
NV_CONTEXT	Yes, for grade check	Set automatically inside Nuvolos Applications.

#### Test your validation script first Before running Grade Check at scale: 1. Open one Student Instance manually. 2. Start the Application you plan to use. 3. Run the exact command you intend to pass to *--command*. 4. Confirm the exit code and output make sense. ### Installation #### Install the CLI `pip install nuvolos-cli` Full documentation is available under [Installing the CLI](/reference/nuvolos-cli-and-python-api/installing-the-cli.md) or at [Nuvolos CLI](https://nuvolos-cli.readthedocs.io/en/latest/). #### Install collect support *grade collect*, and the *collect* step inside grade check, both depend on *nuvolos\_collect*. This package isn’t bundled as a pip extra, due to a dependency conflict between package versions on PyPI, so it must be installed separately: `pip install git+https://github.com/alphacruncher/nv-collect.git` If this package is missing, the CLI will tell you to install it. #### Configure API access If you aren’t already running inside Nuvolos, configure your API key with either of the following: {% code overflow="wrap" %} ``` nuvolos config --api-key export NUVOLOS_API_KEY=... ``` {% endcode %} For additional debug output, you can optionally set: `export NUVOLOS_CLI_DEBUG=true` ### Before you run anything #### Checklist * You’re in the **Master Instance** of the Course. * You know your **Organization slug** (-o) and **Space slug** (-s). * You know the **Application slug** used on Student Instances (-a). * You know the **assignment name** and **assignment folder** used in hand-ins. * Your validation command has already been tested on a sample student. * `nuvolos_collect` is installed, if you plan to collect submissions. * Your API key works (nuvolos instances list -o ... -s ... succeeds). #### Discover your slugs `nuvolos orgs list`\ `nuvolos spaces list -o `\ `nuvolos instances list -o -s `\ `nuvolos apps list -o -s -i master`\ \ *# And on a Student Instance:*\ `nuvolos apps list -o -s -i ` #### Assignment name vs. assignment folder These two flags map to nvcollect conventions under the hand-in tree:


Flag	Meaning	Example
--assignment-name	Bundle/assignment name under handin	homework1
--assignment-folder	Leaf folder name inside each submission	hw1 or submission

If you already use nvcollect or the assignments-review UI, reuse the same names you use there. ## Quick start {% stepper %} {% step %} **Step 1: First run - plan only, no Applications started** {% code overflow="wrap" %} ``` nuvolos grade check \ -o my-university \ -s intro-python-2026 \ -a jupyter \ --assignment-name homework1 \ --assignment-folder hw1 \ --command 'python hw1/check.py' \ --dry-run \ --limit 1 ``` {% endcode %} {% endstep %} {% step %} **Step 2: Real run, one student** {% code overflow="wrap" %} ``` nuvolos grade check \ -o my-university \ -s intro-python-2026 \ -a jupyter \ --assignment-name homework1 \ --assignment-folder hw1 \ --command 'python hw1/check.py' \ --limit 1 \ --continue-on-error ``` {% endcode %} {% endstep %} {% step %} **Step 3: Whole class** {% code overflow="wrap" %} ``` nuvolos grade check \ -o my-university \ -s intro-python-2026 \ -a jupyter \ --assignment-name homework1 \ --assignment-folder hw1 \ --command 'python hw1/check.py' \ --all \ --continue-on-error \ --parallel 3 ``` {% endcode %} {% endstep %} {% endstepper %} {% hint style="info" %} Always run a single-student dry run, then a single-student real run, before scaling up to the whole class. This confirms your flags and validation command are correct before you commit compute time and Credits to a full batch. {% endhint %} #### What success looks like Near the end of the run, you’ll see a summary similar to this: {% code overflow="wrap" %} ``` === Grade run complete === ok=... failed=... skipped=... dry_run=... Durable paths: /files/assignments-review/handin/... and .../handback/... (students see handback, read-only) GRADE CHECK SUMMARY run_id=... { "total": ..., "ok": ..., "failed": ..., "skipped": ..., "dry_run": ... } ``` {% endcode %} The run ID is a UTC timestamp, such as 20260910T143015Z. For each student, the CLI also prints the status, exit code, handin/handback paths, and — often — the full output.log contents. ## Command reference The grade feature is a command group with three subcommands: nuvolos grade\ ├── collect Collect submissions only\ ├── resolve-manifest Preview who would be graded from a manifest\ └── check Full pipeline: collect → test → publish #### Nuvolos grade check (main command) **Purpose**: Collect submissions (optionally), run your validation command on each Student Instance, and publish the results. `nuvolos grade check [OPTIONS]` **Required flags**


Option	Short	Description
--org	-o	Organization slug
--space	-s	Space slug
--app-slug	-a	Application slug to start on each Student Instance
--command	-c	Shell command to run inside each student’s Application

Also required, unless `--skip-collect` is set:


Option	Description
--assignment-name	Assignment/bundle name under handin
--assignment-folder	Leaf folder inside each submission

With `--skip-collect`, provide `--manifest` (or the deprecated `--target-folder`) pointing at an existing collect tree or manifest. **Common optional flags**


Option	Default	Description
--instance / -i	all	Limit to a single Student Instance slug
--limit N	none	Process at most N students
--all	off	Grade every resolved student (clears --limit)
--parallel N	1	Number of concurrent student checks
--continue-on-error	off	Keep going after the first failure
--skip-missing-instances	off	Skip students not visible in the Space
--dry-run	off	Plan only — no start, execute, or stop
--skip-collect	off	Reuse an existing manifest / prior collect
--manifest / -m	—	Manifest path, when skipping collect
--pull-logs / --no-pull-logs	pull on	Pull and publish output.log
--instructor-instance	master (or context)	Where logs are distributed

**Advanced flags**


Option	Notes
--results-dir / -r	Overrides the staging root (default /files/.nuvolos_grade). Durable output still goes to handin/handback.
--target-folder	Deprecated for normal checks; with --skip-collect, can point at an existing collect tree.
--test-command	Deprecated alias for --command.

Exit behavior: If any student fails and the run isn’t a dry run, the CLI exits with an error after printing the summary. Individual validation exit codes are stored in grade\_meta.json and in the summary. #### Nuvolos grade collect **Purpose**: Gather submissions only - no start or execute step. {% code overflow="wrap" %} ``` nuvolos grade collect \ --assignment-name homework1 \ --assignment-folder hw1 \ --target-folder /files/my-collect-run \ -o my-university \ -s intro-python-2026 ``` {% endcode %}


Option	Required	Description
--assignment-name	Yes	Bundle name under handin
--assignment-folder	Yes	Leaf folder name
--target-folder	Yes	Destination for the collected trees and nvcollect_manifest.json
--org / -o	Recommended	Enables renaming folders to student email
--space / -s	Recommended	Same as above

Without --org/--space, folders may stay named by Instance slug instead of student email. Internally, this calls nuvolos\_collect.collect, then optionally renames folders to student email and rewrites the manifest. #### nuvolos grade resolve-manifest **Purpose**: Preview the roster — map manifest rows to Instances and check visibility before a large run. {% code overflow="wrap" %} ``` nuvolos grade resolve-manifest \ -m /path/to/nvcollect_manifest.json \ -o my-university \ -s intro-python-2026 ``` {% endcode %}


Option	Description
--manifest / -m	Manifest file or parent directory
--org / -o, --space / -s	Required
--instance / -i	Optional filter
-f (table or json)	Output format (default table)

Example table columns: instance\_slug | email | name | in\_space | target {% hint style="info" %} Tip: Run this command before any large batch to catch missing Instances or bad filters early. {% endhint %} ## Common workflows {% stepper %} {% step %} **Workflow 1 — Happy path: collect and grade in one go** Use this when hand-ins are already in the review area and you want a single command. {% code overflow="wrap" %} ``` nuvolos grade check \ -o ORG -s SPACE -a APP \ --assignment-name NAME \ --assignment-folder FOLDER \ --command 'python check.py' \ --all \ --continue-on-error ``` {% endcode %} {% endstep %} {% step %} **Workflow 2 — Collect once, grade many times** Useful while you’re still tuning the validation command. {% code overflow="wrap" %} ``` # 1. Collect nuvolos grade collect \ --assignment-name NAME \ --assignment-folder FOLDER \ --target-folder /files/collect-hw1 \ -o ORG -s SPACE # 2. Inspect the roster nuvolos grade resolve-manifest -m /files/collect-hw1 -o ORG -s SPACE # 3. Grade without re-collecting nuvolos grade check \ -o ORG -s SPACE -a APP \ --skip-collect \ --manifest /files/collect-hw1 \ --command 'python check.py' \ --limit 3 \ --continue-on-error ``` {% endcode %} {% endstep %} {% step %} **Workflow 3 — Grade a single student (debug)** {% code overflow="wrap" %} ``` nuvolos grade check \ -o ORG -s SPACE -a APP \ --assignment-name NAME \ --assignment-folder FOLDER \ --command 'python check.py' \ --instance student-instance-slug \ --continue-on-error ``` {% endcode %} {% endstep %} {% step %} **Workflow 4 — Parallel batch with safe defaults** {% code overflow="wrap" %} ``` nuvolos grade check \ -o ORG -s SPACE -a APP \ --assignment-name NAME \ --assignment-folder FOLDER \ --command 'python check.py' \ --all \ --parallel 4 \ --continue-on-error \ --skip-missing-instances ``` {% endcode %} Important: Without --continue-on-error, parallel mode is forced back to 1 so the run can stop cleanly after the first failure. {% endstep %} {% step %} **Workflow 5 — Dry-run capacity planning** {% code overflow="wrap" %} ``` nuvolos grade check \ -o ORG -s SPACE -a APP \ --assignment-name NAME \ --assignment-folder FOLDER \ --command 'python check.py' \ --dry-run \ --all ``` {% endcode %} No Applications are started. This is useful for validating flags and environment gates before a real run. {% endstep %} {% endstepper %} ## Understanding results #### Status values (per student)


Status	Meaning
executed	Lifecycle completed; validation exit code was 0.
failed	Application missing, start/execute error, non-zero exit code, publish failure, timeout, and similar.
skipped	Filtered out, missing Instance (when allowed), or left unprocessed after an earlier failure.
dry_run	Would have run, but --dry-run was set.
pending	Internal/incomplete — should be rare in a final summary.

#### How “ok” is counted * The summary’s ok count is approximately the number of students with status executed. * A non-zero validation exit code marks a student as failed, even if start and stop both worked correctly. * A publish error can flip an otherwise successful execution to failed. #### What students see After a successful publish and handback, students see files under handback, inside `_grading/`. These files are typically read-only and contain: * `output.log` — the command output * `grade_meta.json` — a machine-readable summary #### What instructors see * Live terminal progress and log excerpts. * A final **GRADE CHECK SUMMARY**. * The same artifacts under both handin (next to the submission) and handback. * Staging copies under `/files/.nuvolos_grade///` while the run is active. #### Typical grade\_meta.json fields {% code overflow="wrap" %} ``` { "run_id": "", "instance_slug": "jane-doe", "email": "jane.doe@university.edu", "status": "executed", "error": null, "exit_code": 0, "finished_at": "", "command": "python check.py" } ``` {% endcode %} #### Terminal summary structure For each student, the CLI prints roughly the following: * Email / folder * Instance slug * Status (with exit code) * Error, if any * handin path * handback path (student-visible) * Published file paths * Local log path * Full or truncated output.log A batch handback status appears once at the end of the run. ## Options explained #### **--command / -c** Runs inside the student’s Application, with the working directory effectively under /files. For example: {% code overflow="wrap" %} ``` --command 'python assignments/hw1/test_runner.py' --command 'Rscript -e "testthat::test_dir(\"tests\")"' --command 'bash scripts/validate.sh {instance_slug}' ``` {% endcode %} **Available placeholders** (simple string replacement):


Placeholder	Expands to
{instance_slug}	Student Instance slug
{instance}	Same as {instance_slug}
{target}	Collect target path for that student (may be empty)

**Tips for writing your validation command:** * Prefer simple, non-interactive commands. * Print clear pass/fail messages to stdout. * Set exit codes deliberately (sys.exit(0) / sys.exit(1)). * Avoid relying on GUI applications. #### **--parallel** * Defaults to 1 (sequential). * Higher values start multiple student lifecycles at once. * Requires --continue-on-error to actually run in parallel; otherwise the CLI forces sequential execution. * Start small — 2 to 4 concurrent checks — and monitor cluster capacity and course Credits as you scale up. #### **--continue-on-error**


Without the flag	With the flag
The first hard failure stops the queue; remaining students are marked skipped.	The run keeps going through the full roster.
Parallel execution is forced to 1.	Parallel execution is allowed.

Recommendation: Always pass --continue-on-error for class-wide runs. #### **--limit vs. --all** * \--limit 5 processes the first 5 resolved students — useful for staged rollouts. * \--all processes everyone and clears any limit. #### **--pull-logs / --no-pull-logs** Defaults to on. After execution, logs are distributed to the instructor Instance and published. Turn this off only for special-case debugging — you won’t get the usual handin/handback artifacts. #### **--instructor-instance** Defaults to the Instance from NV\_CONTEXT, falling back to master. Logs are distributed to this Instance so the instructor can read them on the local /files tree. #### Timeouts (environment variables)


Variable	Default	Meaning
GRADE_EXEC_TIMEOUT_SECS	600 (10 min)	Maximum wait for one student’s validation command
GRADE_STAGE_TIMEOUT_SECS	120	Maximum wait while staging logs before distribution
APP_START_TIMEOUT_SECS	600	Application start wait (shared across the CLI)
APP_TASK_TIMEOUT_SECS	600	Background task wait, e.g. distribute

export GRADE\_EXEC\_TIMEOUT\_SECS=1200 *# 20 minutes, for slow tests* ## File layout and artifacts #### Staging layout (scratch) /files/.nuvolos\_grade/\ └── \/ e.g. 20260910T143015Z\ ├── nvcollect\_manifest.json (after collect)\ ├── /\ │ ├── ...collected files...\ │ ├── output.log\ │ ├── .cmd\_done\ │ └── .cmd\_exit.0\ └── /\ └── ... Folders are preferably named by student email, when org/Space resolution succeeds. #### Durable layout (what matters long-term) /files/assignments-review/\ ├── handin/\ │ └── \/\/\\_.../\ │ ├── ...submission files...\ │ └── \_grading/\ │ ├── output.log\ │ └── grade\_meta.json\ └── handback/\ └── \/\/\\_.../\ └── \_grading/\ ├── output.log ← student-visible, read-only\ └── grade\_meta.json #### Why the \_grading/ subfolder? So grading files never collide with a student’s own output.log, or other similarly named files, inside their submission. ## Rules and limits #### Hard gates (the command will refuse to run without these) * An API key must be configured. * NV\_CONTEXT must match -o / -s. * The context Instance must be the Master Instance. * The Space type must be Course. #### Operational safety


Concern	Behavior
Applications left running	Stop is always attempted in a finally block after start.
One student fails	Without --continue-on-error, remaining students are skipped.
Missing Application on an Instance	That student is marked failed; others may continue if allowed.
Missing Instance in the Space	Errors, or is skipped with --skip-missing-instances / --continue-on-error.
Credits and cost	Each Application start consumes Nuvolos Compute Units (NCUs) and draws down your Space’s Credits — prefer --limit first, then scale up.
Student-facing feedback	Handback files are made read-only on a best-effort basis.

{% hint style="danger" %} #### What Grade Check does not guarantee yet * Automatic Credit protection or billing controls for student budgets — this is a platform-level concern and should be planned separately. * GPU-aware scheduling beyond what the Application start API already supports. * Keeping Applications running for external, interactive evaluation — that’s a different feature (see *Related but separate: evaluation sessions*). {% endhint %} ## Troubleshooting | Error | Description | | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *“must run from the master instance”* | You’re not on the Master Instance, or NV\_CONTEXT points elsewhere. Open a terminal on the teaching Space’s Master Instance. | | *“only supported in teaching spaces”* | Wrong Space type. Use the course teaching Space, not a research Space. | | *“nuvolos\_collect is required”* | Install: pip install git+ | | *“App slug … not found on instance”* | That Student Instance doesn’t have the Application installed. Distribute the course Application to all Instances, or filter with --instance, or fix the image. | | *“Timed out … waiting for command completion”* |

The validation command ran longer than GRADE\_EXEC\_TIMEOUT\_SECS (10 minutes by default), or the done-marker never appeared.

Test the command manually on one student.
Increase GRADE\_EXEC\_TIMEOUT\_SECS.
Make sure the command is non-interactive and eventually exits.

| | *“Instance not found … (or API key lacks access)”* | The manifest references a student your API key can’t see. Use --skip-missing-instances for partial runs, or fix access/roster. | | Parallel execution didn’t parallelize | You forgot --continue-on-error. Without it, workers are forced to 1. | | No logs, or an empty handback |

Confirm --pull-logs is enabled (it’s on by default).
Check whether execution actually produced output.log.
Look at staging under /files/.nuvolos\_grade/\/....
Confirm distribute permissions and the instructor Instance are correct.

| | Collect folder names are slugs, not emails | Pass -o and -s (or rely on NV\_CONTEXT) so the email relabeling step can run. | | Command failed, but I need the log | Even on failure, Grade Check tries to pull logs before stopping the Application. Check the summary’s log and handin paths, plus the terminal excerpt. | | The whole CLI exits non-zero after partial success | This is by design: if failed > 0, grade check raises an error at the end so CI pipelines and scripts notice. Read the printed summary — individual students may still be fine. |