Run a TAO training/evaluation/inference container on an NVIDIA Brev GPU instance. Instance provisioning (create/search/stop/delete/login) is delegated to the official brev-cli agent skill or the Brev MCP server; this skill covers only the TAO-specific part — running the container over `brev exec` via the four-verb docker contract. Trigger phrases include "run on Brev", "Brev GPU instance", "TAO on Brev", "submit job to Brev".
Intro in this page language first. The official description stays in its original wording; we do not rewrite SKILL.md.
What it does
Run a TAO training/evaluation/inference container on an NVIDIA Brev GPU instance. Instance provisioning (create/search/stop/delete/login) is delegated to the official brev-cli agent skill or the Brev MCP server; this skill covers only the TAO-specific part — running the container over `brev exec` via the four-verb docker contract. Trigger phrases include "run on Brev", "Brev GPU instance", "TAO on Brev", "submit job to Brev".
When to use it
The official description does not include a separate “Use when”. Per the spec, agents activate this skill when the task matches keywords in that description.
How agents load it
Per Agent Skills progressive disclosure: name and description load at startup (~100 tokens); the full SKILL.md body loads when the skill activates; scripts/, references/, and assets/ load only as needed. This file's sections: Brev — TAO execution glue; Provisioning: use the official Brev skill or MCP; installs to ~/.claude/skills/brev-cli/ , ~/.codex/skills/brev-cli/ , ~/.agents/skills/brev-cli/; Storage; Execution — the four verbs (a compound over Docker); `brev exec` argument form.
File analysis
File analysis: besides SKILL.md, the body references scripts/install-agent-skill.sh, scripts/tao_job_record.py. Those resources load on demand.
Brev — TAO execution glueProvisioning: use the official Brev skill or MCPinstalls to ~/.claude/skills/brev-cli/ , ~/.codex/skills/brev-cli/ , ~/.agents/skills/brev-cli/StorageExecution — the four verbs (a compound over Docker)`brev exec` argument formNGC auth (one-time per instance) — value never on argv.Single-quoted locally so $NGC_KEY expands in the instance's shell; export itthere first (or pipe it in from the local shell, if the instance has no copy).Verify auth without reading ~/.docker/config.json. Failure before a successfullogin = not authenticated; failure after = the key's org lacks entitlement.Pull BEFORE the GPU run. `docker run` would pull implicitly, but the instance
Compatibility:Requires the brev CLI (https://github.com/brevdev/brev-cli) and an active brev login. Instance provisioning is handled by the official brev-cli agent skill or the Brev MCP server.
· License:Apache-2.0
· allowed-tools:Read Bash
Run a TAO training/evaluation/inference container on an NVIDIA Brev GPU instance. Instance provisioning (create/search/stop/delete/login) is delegated to the official brev-cli agent skill or the Brev MCP server; this skill covers only the TAO-specific part — running the container over `brev exec` via the four-verb docker contract. Trigger phrases include "run on Brev", "Brev GPU instance", "TAO on Brev", "submit job to Brev".
compatibility
Requires the brev CLI (https://github.com/brevdev/brev-cli) and an active brev login. Instance provisioning is handled by the official brev-cli agent skill or the Brev MCP server.
allowed-tools
Read BashExperimental field; support depends on the client and does not grant permissions by itself.
License
Apache-2.0
DiscoverThe client exposes names and descriptions to the agent.
ActivateYour request or the task context selects the skill and loads its instructions.
Load resourcesReferenced scripts, documentation and assets are used when needed.
Files referenced by the instructions · 2
scripts/install-agent-skill.sh
scripts/tao_job_record.py
These paths are extracted from the text. Check the upstream package to verify the files exist.
Choose the target agent and installation scope, keep referenced package files, then verify the skill appears in the client's catalog.
This skill references supporting files. Retrieve the complete directory from the source; copying SKILL.md alone may leave missing dependencies.
Ask your Agent to install
Copy these instructions to a compatible agent and confirm the target directory matches your client.
Install the agent skill "tao-run-on-brev" into my project. The full SKILL.md and official description are at https://zicq.com/en/skills/skl-abc9363b952babd2-Tao-Run-On-Brev.html
Save it as .cursor/skills/tao-run-on-brev/SKILL.md or .claude/skills/tao-run-on-brev/SKILL.md and keep the frontmatter name and description exactly as-is.
This skill also ships scripts/, references/, or assets/ — fetch the whole folder from https://github.com/nvidia/skills instead of creating only a SKILL.md.
The CLI lets you choose the agent interactively. The default scope is the project; use -g for user scope. Confirm package availability with the discovery command, then use npx skills list to inspect installed skills.
---
name: tao-run-on-brev
description: Run a TAO training/evaluation/inference container on an NVIDIA Brev GPU instance. Instance provisioning (create/search/stop/delete/login) is delegated to the official brev-cli agent skill or the Brev MCP server; this skill covers only the TAO-specific part — running the container over `brev exec` via the four-verb docker contract. Trigger phrases include "run on Brev", "Brev GPU instance", "TAO on Brev", "submit job to Brev".
license: Apache-2.0
compatibility: Requires the brev CLI (https://github.com/brevdev/brev-cli) and an active brev login. Instance provisioning is handled by the official brev-cli agent skill or the Brev MCP server.
metadata:
author: NVIDIA Corporation
version: "0.2.0"
allowed-tools: Read Bash
tags:
- gpu
- compute
- instance-based
- brev
---
# Brev — TAO execution glue
> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).
NVIDIA Brev provides on-demand GPU instances (pre-loaded with NVIDIA drivers,
CUDA, Docker, and the NVIDIA Container Toolkit). Brev is **instance-based**: you
provision an instance, run commands on it over `brev exec`, and delete it when
done.
This skill is deliberately thin. **Provisioning and managing instances — create,
search by GPU/price, start/stop, delete, login — is owned by NVIDIA Brev's own
agent skill, not duplicated here.** This skill covers only the TAO-specific part:
running a TAO container on a reached instance through the **four-verb docker
contract**, deferring the container-how to `tao-run-on-docker` over `brev exec`.
## Provisioning: use the official Brev skill or MCP
NVIDIA Brev publishes an agent skill that manages instances in natural language
("create an A100 instance", "search for GPUs under $3/hr", "stop all my
instances"). Install it once — it self-registers into your agent's skills dir and
is discovered at runtime:
```bash
curl -fsSL https://raw.githubusercontent.com/brevdev/brev-cli/main/scripts/install-agent-skill.sh | bash
# installs to ~/.claude/skills/brev-cli/ , ~/.codex/skills/brev-cli/ , ~/.agents/skills/brev-cli/
```
Or connect the **Brev MCP server** (`https://docs.nvidia.com/brev/_mcp/server`).
Either one owns login/auth quirks, placement IDs, GPU search, and teardown flags.
It does **not** cover container execution on the instance — that is this skill.
**Preflight for this skill:** the `brev` CLI is on `PATH` and logged in (headless:
`brev login --token "$BREV_API_TOKEN"` before any other call), and you can reach a
target instance — poll with a **two-word** command until it succeeds before
issuing real work (a fresh instance reports `RUNNING` before sshd is up):
```bash
for i in $(seq 1 60); do brev exec "echo ok" 2>/dev/null | grep -qx ok && break; sleep 5; done
brev exec "echo ok" 2>/dev/null | grep -qx ok || { echo "instance not exec-ready"; exit 1; }
```
The probe must be **two words, quoted as one argument**. A single-token probe
(`brev exec -- true`) passes even when every real command is broken,
because `brev exec [instance...] ` treats only the LAST positional as
the command — so a lone `true` lands in the right slot by accident while
`docker run ...` does not. See *`brev exec` argument form* below.
Allow **≥ 600 s** for the first `brev exec` on a new instance (SSH bring-up +
first container pull); a 60–120 s wrapper timeout truncates startup and looks
like a spurious `exec failed`.
## Storage
No shared NFS/Lustre — storage tier **B/C** via `tao-data-io`: stage inputs from
S3 to the instance's local disk (or fetch in-container) and **upload results to S3
before deleting the instance**. Instance-local `~/` persists across stop/start but
**not** across delete/create, so the results upload must precede teardown.
## Execution — the four verbs (a compound over Docker)
Brev is a **compound consumer**: `submit` reaches an instance, then **defers the
container-how to the four docker verbs** (`tao-run-on-docker`) run over
`brev exec`. It is not a symmetric peer — teardown must additionally delete the
instance to stop billing. `$BANK` = `${TAO_SKILL_BANK_PATH}`.
- **submit** — reach an instance (provision/reuse via the official Brev skill or
MCP; reuse an existing instance by its `instance_id`; wait for readiness, above).
Lint the assembled command, open the record to mint `$JOB_ID` **before** launch,
then run the docker `submit` verb *inside* the instance and mark RUNNING:
```bash
redact_secrets.py lint <<<"$REMOTE_CMD" # no inline secrets; creds as -e VAR
JOB_ID=$("$BANK/scripts/tao_job_record.py" open \
--platform brev --image "$IMG" \
--network-arch "$ARCH" --action "$ACTION" \
--storage-tier "$TIER" --results-root "$RESULTS_ROOT")
brev exec "docker inspect '$JOB_ID' >/dev/null 2>&1 && { echo '$JOB_ID already submitted'; exit 0; }; docker run -d --name '$JOB_ID' --label 'tao-job=$JOB_ID' ..."
"$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state RUNNING \
--backend-ref "/$JOB_ID" # instance is part of the ref: the
# container is unreachable without it
```
- **status / logs** — `brev exec "docker inspect $JOB_ID"` /
`brev exec "docker logs $JOB_ID"`, mapped to the vocab exactly as
the docker verbs do. Recover `` from the record's `backend-ref`.
- **cancel / teardown** — remove the container, then for an ephemeral instance
delete it (stops billing), then mark the record. Never leave an ephemeral
instance running:
```bash
brev exec "docker rm -f $JOB_ID"
brev delete # ephemeral instances only
"$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state CANCELED --source agent
```
### `brev exec` argument form
The remote command is **one argument**. The CLI signature is
`brev exec [instance...] `: every positional except the last is an
instance name, and `--` only ends flag parsing — it does not group the words
after it. So `brev exec -- docker inspect "$JOB_ID"` is read as
instances ` docker inspect` plus command `"$JOB_ID"`, and fails with
`could not look up instance "docker"` / `ssh: illegal option -- -` (exit 255) —
an error that reads like an instance or SSH fault but is a syntax fault. Quote
every remote command as a single string, exactly as `brev exec --help` shows.
NGC auth once per instance — **never put `NGC_KEY` on argv** (it lands in the
remote process table); pipe it to `--password-stdin`:
```bash
IMG=nvcr.io/nvidia/tao/tao-toolkit:7.2.0-pyt # versions-key: images.tao_toolkit.pyt
# NGC auth (one-time per instance) — value never on argv.
# Single-quoted locally so $NGC_KEY expands in the instance's shell; export it
# there first (or pipe it in from the local shell, if the instance has no copy).
brev exec 'printf %s "$NGC_KEY" | docker login nvcr.io -u "$oauthtoken" --password-stdin'
# Verify auth without reading ~/.docker/config.json. Failure before a successful
# login = not authenticated; failure after = the key's org lacks entitlement.
brev exec "docker manifest inspect $IMG >/dev/null && echo AUTH_OK || echo AUTH_FAIL"
# Pull BEFORE the GPU run. `docker run` would pull implicitly, but the instance
# bills from boot, so a multi-GB first-time TAO pull is billed GPU-idle time.
# Pulling as its own step also separates a pull failure (auth/entitlement) from
# a training failure in the logs.
brev exec "docker image inspect $IMG >/dev/null 2>&1 || docker pull $IMG"
# Run a TAO job (the docker `submit` verb, over brev exec)
brev exec "docker inspect '$JOB_ID' >/dev/null 2>&1 && { echo '$JOB_ID already submitted'; exit 0; }; docker run -d --name '$JOB_ID' --label 'tao-job=$JOB_ID' --gpus all -v ~/data:/data -e NGC_KEY '$IMG' visual_changenet train -e /data/spec.yaml"
```
## Multi-GPU and multi-node
**Multi-node is not supported on Brev** — instance-based, no cross-instance
coordination. Multi-GPU **on a single instance** is supported (up to 8× H100 /
A100 / L40S); `torchrun --nproc-per-node=N` or PyTorch DDP work within the
instance.