Skip to main content
ZICQ

Skills ZICQ category:Media physical-ai-video-data-augmentation

Physical Ai Video Data Augmentation

Use when running video data augmentation and auto-labeling workflows on OSMO: flow selection, preflight, submit-time interpolation, monitoring, and output retrieval. Trigger keywords: video data augmentation, data enrichment, auto labeling, VDA demo, OSMO workflow, pseudo labeling.

1933 installs

Official URL:skills.sh

What this skill does

Intro in this page language first. The official description stays in its original wording; we do not rewrite SKILL.md.

What it does

Use when running video data augmentation and auto-labeling workflows on OSMO: flow selection, preflight, submit-time interpolation, monitoring, and output retrieval. Trigger keywords: video data augmentation, data enrichment, auto labeling, VDA demo, OSMO workflow, pseudo labeling.

When to use it

running video data augmentation and auto-labeling workflows on OSMO: flow selection, preflight, submit-time interpolation, monitoring, and output retrieval. Trigger keywords: video data augmentation, data enrichment, auto labeling, VDA demo, OSMO workflow, pseudo labeling

How agents load it

Per Agent Skills progressive disclosure: name and description load at startup (~100 tokens); the full SKILL.md body loads when the skill activates; scripts/, references/, and assets/ load only as needed. This file's sections: Physical AI Video Data Augmentation Workflow Orchestrator; Purpose; Prerequisites; Instructions; Available Scripts; Supported Flows. It includes spec-recommended sections: step-by-step instructions.

File analysis

File analysis: besides SKILL.md, the body references scripts/preflight_credentials.sh, assets/configs/osmo/augmentation_and_al.yaml, scripts/pre_submit_guard.py, assets/configs/osmo/auto_labeling.yaml, scripts/prepare_demo_assets.sh, scripts/generate_configs.py. Those resources load on demand.

Physical AI Video Data Augmentation Workflow OrchestratorPurposePrerequisitesInstructionsAvailable ScriptsSupported FlowsPick the right workflow for the user's requestDisambiguation: handle vague requests before committingAutonomous defaults (do NOT ask)Triggers that should pause for disambiguationWhen NOT to disambiguateStep 0: Select Flow and Gather Inputs

· License:CC-BY-4.0 AND Apache-2.0

Source category:skills.sh agent-skill

SKILL.md & Agent activation

Official spec ↗
name
physical-ai-video-data-augmentation
description
Use when running video data augmentation and auto-labeling workflows on OSMO: flow selection, preflight, submit-time interpolation, monitoring, and output retrieval. Trigger keywords: video data augmentation, data enrichment, auto labeling, VDA demo, OSMO workflow, pseudo labeling.
License
CC-BY-4.0 AND Apache-2.0
  1. DiscoverThe client exposes names and descriptions to the agent.
  2. ActivateYour request or the task context selects the skill and loads its instructions.
  3. Load resourcesReferenced scripts, documentation and assets are used when needed.
Files referenced by the instructions · 8
  • scripts/preflight_credentials.sh
  • assets/configs/osmo/augmentation_and_al.yaml
  • scripts/pre_submit_guard.py
  • assets/configs/osmo/auto_labeling.yaml
  • scripts/prepare_demo_assets.sh
  • scripts/generate_configs.py
  • scripts/cosmos_worker.sh
  • scripts/pl_original_worker.sh

These paths are extracted from the text. Check the upstream package to verify the files exist.

Invocation syntax and available tools depend on your Agent client. Client integration guide ↗

Install this skill

Skills CLI ↗

Choose the target agent and installation scope, keep referenced package files, then verify the skill appears in the client's catalog.

This skill references supporting files. Retrieve the complete directory from the source; copying SKILL.md alone may leave missing dependencies.

Ask your Agent to install

Copy these instructions to a compatible agent and confirm the target directory matches your client.

Install the agent skill "physical-ai-video-data-augmentation" into my project. The full SKILL.md and official description are at https://zicq.com/en/skills/skl-6ac17ac353cd8d61-Physical-Ai-Video-Data-Augmentation.html
Save it as .cursor/skills/physical-ai-video-data-augmentation/SKILL.md or .claude/skills/physical-ai-video-data-augmentation/SKILL.md and keep the frontmatter name and description exactly as-is.
This skill also ships scripts/, references/, or assets/ — fetch the whole folder from https://github.com/nvidia/skills instead of creating only a SKILL.md.

Full package on GitHub ↗

Install from the terminal · Skills CLI

Requires Node.js and npx. First inspect the repository's skill list to confirm the name.

npx skills add 'https://github.com/nvidia/skills' --list

npx skills add 'https://github.com/nvidia/skills' --skill 'physical-ai-video-data-augmentation'

The CLI lets you choose the agent interactively. The default scope is the project; use -g for user scope. Confirm package availability with the discovery command, then use npx skills list to inspect installed skills.

Readable layout
--- name: physical-ai-video-data-augmentation description: >- Use when running video data augmentation and auto-labeling workflows on OSMO: flow selection, preflight, submit-time interpolation, monitoring, and output retrieval. Trigger keywords: video data augmentation, data enrichment, auto labeling, VDA demo, OSMO workflow, pseudo labeling. license: CC-BY-4.0 AND Apache-2.0 metadata: owner: NVIDIA service: data version: 1.0.0 reviewed: '2026-05-26' author: NVIDIA tags: - physical-ai - video-data-augmentation - auto-labeling - cosmos --- # Physical AI Video Data Augmentation Workflow Orchestrator Default workflow skill for VDA execution on OSMO. It owns flow selection, preflight, cache readiness, inference-path decisions, submit-time interpolation, monitoring, and output retrieval. Component skills are consult-only. ## Purpose Run the end-to-end VDA workflow safely and reproducibly from preflight to output download. Do NOT use this skill for container-internal tuning-only questions. ## Prerequisites Confirm these before running preflight or any submit. Missing required secrets surface as `USER_INPUT_REQUIRED:` from `scripts/preflight_credentials.sh`. | Requirement | How it is satisfied | Used for | |---|---|---| | NGC API key (optional) | `NGC_API_KEY`, `NGC_CLI_API_KEY`, or compatible `nvapi-*` token in `NVIDIA_API_KEY`/`OPENAI_API_KEY`/`VLM_API_KEY`/`LLM_API_KEY` | Optional for `nvcr_io` credential refresh and NGC REST scope probe; default VDA image refs are validated via workflow registry probes | | Hugging Face token | `HF_TOKEN` (or `HUGGING_FACE_HUB_TOKEN`), or a cached token at `~/.cache/huggingface/token` | Creates the OSMO `hf_token` credential; pulls gated Cosmos/SeedVR weights | | OSMO CLI access | `osmo` on `PATH`, logged in, with a default profile and a registered DATA credential profile matching `storage_url` | Submitting/monitoring workflows and listing/downloading objects | | GPU pool | At least one `ONLINE` pool in `osmo pool list --mode free`; `POD_TEMPLATE` carries GPU toleration/selectors | Scheduling setup + worker tasks | Optional (only for the strict NGC org/team probe): `NGC_ORG` + `NGC_TEAM` (or `NGC_CLI_ORG` / `NGC_CLI_TEAM`). External VLM/LLM endpoint keys are validated separately, not by preflight. Key handling rule: `nvapi-*` tokens are first-class inputs for `nvcr_io`. Never reject by token prefix alone; use workflow registry probe results as source of truth. ## Instructions 1. Select the workflow (`auto_labeling`, `augmentation_and_al`, `e2e`, `e2e_super_resolution`) from user intent. 2. Provide a tentative execution-time overview before starting run actions. 3. Run preflight and readiness checks before submit. 4. Derive submit-time values from the active dataset backend (never guess `storage_url`). 5. Submit the workflow with explicit interpolation values and monitor to completion. 6. Retrieve outputs, provide side-by-side comparison evidence for augmented flows, and summarize task outcomes. Use `run_script(...)` for script execution. Canonical examples: ```python run_script("bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/augmentation_and_al.yaml") run_script("python3 scripts/pre_submit_guard.py --workflow assets/configs/osmo/auto_labeling.yaml") run_script("bash scripts/prepare_demo_assets.sh /srv/sdg/data/vda_inputs") ``` ## Available Scripts Use script-level `--help` for exact arguments. | Script | Role | |---|---| | `scripts/preflight_credentials.sh` | Secrets/control-plane preflight and workflow image access checks | | `scripts/pre_submit_guard.py` | Submit-time interpolation, cache, and dataset safety checks | | `scripts/prepare_demo_assets.sh` | Demo video pull + flatten for default demo path | | `scripts/generate_configs.py` | Setup-time config and cookbook projection generation | | `scripts/cosmos_worker.sh` | Augmentation worker execution | | `scripts/pl_original_worker.sh` | Original-video auto-labeling worker execution | | `scripts/pl_augmented_worker.sh` | Augmented-video auto-labeling worker execution | | `scripts/osmo_barrier.py` | Multi-node barrier synchronization | | `scripts/stage_run_artifacts.sh` | Local mirror of full run output + input video | | `scripts/render_side_by_side.sh` | Side-by-side comparison render from local artifacts | ## Supported Flows | Flow | OSMO YAML | Group sequence | Typical use | |---|---|---|---| | `augmentation_and_al` | `assets/configs/osmo/augmentation_and_al.yaml` | setup -> augmentation -> auto_labeling_augmented | Augment one or more videos, then auto-label augmented outputs | | `auto_labeling` | `assets/configs/osmo/auto_labeling.yaml` | setup -> auto_labeling | Label original videos only | | `e2e` | `assets/configs/osmo/e2e.yaml` | setup -> (auto_labeling_original + augmentation) -> auto_labeling_augmented | Throughput-first path | | `e2e_super_resolution` | `assets/configs/osmo/e2e_super_resolution.yaml` | setup -> auto_labeling_original -> augmentation -> auto_labeling_augmented | Sequential path with SR gate before augmentation | Legacy alias `assets/configs/osmo/augmentation_and_pl.yaml` remains for backwards compatibility. ### Pick the right workflow for the user's request | User intent | Workflow | |---|---| | "Label my source videos" / "PL-only" / "no augmentation" | `auto_labeling` | | "Create augmented videos and label them" | `augmentation_and_al` | | "Run the full pipeline quickly" | `e2e` | | "Run full pipeline, but gate on SR-enhanced originals first" | `e2e_super_resolution` | ## Disambiguation: handle vague requests before committing Default to autonomy: ask only when missing information blocks execution. ### Autonomous defaults (do NOT ask) - If dataset source is absent, run VDA demo path (`scripts/prepare_demo_assets.sh`) and continue with `dataset=vda-demo`. - If flow is not explicitly requested, default to `augmentation_and_al`. - If endpoint mode is unspecified, default to in-cluster persistent NIM reuse and automatic NIM deploy/repair when unhealthy. - If cache is missing, run `setup_model_cache.yaml`, rerun pre-submit guard, and continue automatically on success. - After any stage completes successfully, continue to the next stage immediately. Do not pause with "Ready when you are" or equivalent approval prompts. ### Triggers that should pause for disambiguation | Missing input | Why it matters | Ask | |---|---|---| | `USER_INPUT_REQUIRED` from preflight | Required secret is missing | Ask one concise unblock question for exactly the missing value(s) | | Storage backend prefix cannot be derived from the active dataset/upload root | Wrong scheme causes runtime storage auth mismatch | "What is the backend-native root prefix for this run?" | | No ONLINE GPU pool/platform can be selected | Workflow cannot schedule setup/workers | "Which GPU pool/platform should this run target?" | ### When NOT to disambiguate - Do not ask for cookbook unless user explicitly asks to change scene profile. - Do not offer external endpoints by default. - Do not ask A/B cache strategy questions; default is automatic cache setup. - Do not ask to scale down existing NIMs; this is forbidden. - Do not invent, scrape, or generate random videos when input is missing. - Do not use non-VDA demo sources (for example Carline adaptation assets) unless the user explicitly requests a different dataset. ## Step 0: Select Flow and Gather Inputs ### Input video policy (non-negotiable) - Always preserve user-provided video inputs (dataset URL, local path, or upload folder) as first-class and preferred. - Never replace an explicit user video with demo assets or any other source. - If no video input is provided, default to VDA demo assets via `scripts/prepare_demo_assets.sh` (HF dataset flow) without asking extra source-selection questions. - If the user explicitly mentions an input video or dataset, prefer and use that input instead of demo assets. - Use only VDA demo assets (`nvidia/video-data-augmentation-demo`) for the default demo path. - Never propose arbitrary web clip downloads or placeholder videos unless the user explicitly requests that behavior. Collect only missing values: 1. Dataset source (prefer explicit user-provided `dataset_url` or local upload folder; otherwise default to VDA demo assets and proceed). 2. Flow (`auto_labeling`, `augmentation_and_al`, `e2e`, `e2e_super_resolution`); default to `augmentation_and_al` when unspecified. 3. OSMO `gpu_platform` for all VDA resources (auto-select an ONLINE platform when unambiguous; ask only when no valid option exists). 4. Endpoint mode (default in-cluster NIM reuse/deploy unless explicitly overridden). Do not guess `gpu_platform` (for example `microk8s`). Use the exact current platform label shown by `osmo pool list --mode free` (for example `gpu`). Generate run stamp before each submit: ```bash STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8) RUN_ID="run-$STAMP" ``` ## Execution Time Overview (required before run) Before running any mutating command (`osmo credential set`, NIM install/repair, cache workflow submit, or target VDA workflow submit), provide a short ETA overview to the user. Keep it concise (one short paragraph or 4-6 bullets) and include: - whether this looks like a **cold start** (NIM/cache missing) or **warm start** (NIM/cache already healthy), - major phases with approximate durations, - a total expected range for the selected workflow. Baseline ranges (from observed MicroK8s + OSMO runs): | Phase | Typical duration | |---|---| | Credentials + preflight | ~1-2 min | | NIM deploy/download/warmup (if needed) | ~10-15 min | | Demo assets download/upload (if demo path) | ~1-3 min | | Model cache population (if needed) | ~15-25 min | | Workflow submit + queue/start | ~1-3 min | Workflow runtime ranges after submit: | Flow | Typical runtime | |---|---| | `auto_labeling` | ~6-15 min | | `augmentation_and_al` | ~20-35 min | | `e2e` | ~22-40 min | | `e2e_super_resolution` | ~25-45 min | Cold-start end-to-end runs are commonly ~45-80 min; warm-start runs are usually ~20-45 min depending on flow and video length. ## Common Preconditions (all flows) 1. **Credential and control-plane preflight** ```bash bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/.yaml ``` Restricted egress: ```bash bash scripts/preflight_credentials.sh --no-probe --workflow assets/configs/osmo/.yaml ``` Preflight does not require a workload-local `.env`. Runtime interpolation is driven by submit-time values (`dataset`, `run_id`, `gpu_platform`, `video`, `storage_url`, `skills_dir`) supplied in one `--set-string` list. Passing `--workflow` validates pull access for the active workflow image refs (`workflow.groups[].tasks[].image`) using anonymous bearer access with credential fallback when provided. If replacement NGC/HF secrets are provided in env, preflight refreshes existing `nvcr_io` / `hf_token` automatically when present. Use `--refresh` to force overwrite even when no new env secrets were supplied: ```bash bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/.yaml --refresh ``` If output contains `USER_INPUT_REQUIRED:`, ask one concise unblock question and stop. On workflow image `401/403`, report registry access failure after probe checks on the listed image refs; do not claim a key family (for example `nvapi-*`) is categorically unsupported. 2. **Storage interpolation policy** `storage_url` must be derived from the actual dataset/upload backend for the current run. ```text dataset_url=azure://storiondevxah69/osmo-workflows/datasets/vda-demo storage_url=azure://storiondevxah69/osmo-workflows dataset=vda-demo ``` Never silently default to stale `s3://` values on non-S3 backends. 3. **Inference policy (non-negotiable)** - Reuse healthy in-cluster persistent NIM endpoints by default. - If missing/unhealthy, deploy automatically — this is a prerequisite, not a user decision. Do NOT pause to ask; run the install with the VDA allow-list: ```bash export NIM_SERVICES="qwen3-vl qwen25-14b" skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/inference-nim-operator/scripts/install.sh ``` - See `references/nim/README.md` for full endpoint docs and health checks. - External endpoints are opt-in only (explicit request or explicit URLs); only then skip the in-cluster deploy. - Never infer external mode from credential presence. - Never scale down/delete existing NIMs to free GPUs. 4. **Readiness guard** ```bash osmo pool list --mode free osmo config show POD_TEMPLATE python3 scripts/pre_submit_guard.py --workflow assets/configs/osmo/.yaml ``` 5. **Cache auto-remediation** If `pre_submit_guard.py` reports cache failure, default action is to run: ```bash osmo workflow submit assets/configs/osmo/setup_model_cache.yaml \ --set-string storage_url= path=data ``` Then rerun `pre_submit_guard.py` and submit the target VDA flow only after it passes. Ask user only when backend/prefix is ambiguous or cache setup fails. 6. **Scheduling policy** VDA templates schedule setup and workers on `gpu_platform` (no `system` pool dependency for user workloads). ## Submit (all flows) Every flow uses the same submit shape; only the workflow YAML changes. Choose the YAML for the requested flow, then run the command below. Full per-flow walkthroughs (stage matrix and flow details) live in the linked references. | Flow | Workflow YAML | Walkthrough | |---|---|---| | Augmentation + auto-labeling | `assets/configs/osmo/augmentation_and_al.yaml` | `references/flows/augmentation_and_al.md` | | Auto-labeling only | `assets/configs/osmo/auto_labeling.yaml` | `references/flows/auto_labeling.md` | | E2E (parallel) | `assets/configs/osmo/e2e.yaml` | `references/flows/e2e.md` | | E2E (super-resolution gated) | `assets/configs/osmo/e2e_super_resolution.yaml` | `references/flows/e2e_super_resolution.md` | ```bash SKILLS_DIR="$(cd "$(git rev-parse --show-toplevel)/skills/physical-ai-video-data-augmentation" && pwd)" STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8) osmo workflow submit assets/configs/osmo/.yaml \ --pool \ --set-string \ dataset= \ run_id=run-$STAMP \ storage_url= \ gpu_platform= \ video= \ cosmos_model_cache_url=/data/models/cosmos_transfer \ auto_labeling_model_cache_url=/data/models/auto_labeling \ skills_dir="$SKILLS_DIR" ``` Compatibility note: - Use exactly one `--set-string` flag and pass all the key/value pairs after it. - Do not repeat `--set`/`--set-string` flags in the same command; some OSMO builds only honor the last occurrence. - Do not mix `--set` and `--set-string` in one submit command. - Pass explicit `*_model_cache_url` values to avoid nested-template interpolation differences across OSMO environments. - Do not brute-force permutations of flags. Use this shape directly. Common optional overrides (append key/value pairs to the same `--set-string` list): ```bash cookbook= \ vlm_url= \ llm_url= \ cosmos_model_cache_url= \ auto_labeling_model_cache_url= ``` The auto-labeling-only flow has no augmentation stage, so it omits `cosmos_model_cache_url` at runtime; passing it is harmless and keeps one submit shape across flows. ## OSMO Monitoring ```bash # Workflow status + task states osmo workflow query --format-type json \ | jq '{status, tasks: [.groups[].tasks[] | {name, status, exit_code}]}' # Logs for a specific task osmo workflow logs --task -n 200 # Output retrieval osmo data list --no-pager osmo data download / ``` For completion artifacts, always mirror the full run output into workspace: ```bash ROOT="$(git rev-parse --show-toplevel)" RUN_LOCAL_DIR="$ROOT/media/vda/runs/" mkdir -p "$RUN_LOCAL_DIR" osmo data download "/datasets/-outputs//" "$RUN_LOCAL_DIR/" ``` For runs expected to exceed two minutes, send heartbeat updates at least every two minutes. For media evidence, emit one standalone `MEDIA:` line per message bubble. Execution continuity requirement: - Heartbeats must report progress while continuing work; they are status updates, not permission prompts. - Do not stop between green stages waiting for approval. - Pause only on blocking failures or explicit user stop/redirect. - If submit fails on interpolation, rerun once with the same canonical single-flag shape and corrected values; do not loop through ad-hoc flag experiments. MEDIA formatting is strict: - Emit exactly one line: `MEDIA:/absolute/path/to/file.mp4` - Keep `MEDIA:` contiguous on a single line (never split across lines). - No extra text in the same bubble. - No code fences, bullets, or quotes around the directive. - If render fails: retry once from a stable workspace path, then emit PNG fallback. ## Post-Run Comparison Evidence (required for augmented flows) Applies to `augmentation_and_al`, `e2e`, and `e2e_super_resolution` after a successful run. Required completion output (do not stop at raw output URLs): 1. Stage full outputs + input video into workspace-local path: ```bash bash scripts/stage_run_artifacts.sh \ --storage-url --dataset --run-id --video ``` 2. Render side-by-side from that local run copy: ```bash bash scripts/render_side_by_side.sh \ --run-local-dir "/media/vda/runs/" --dataset --video ``` 3. Emit MEDIA from the local run copy and include: - augmentation summary from `/setup_b0/configs/manifest.yaml` (`sampled_vars` for `_aug0`) - auto-labeling summary from `/outputs/pseudo_labeled_augmented/_aug0` - for `e2e` / `e2e_super_resolution`, original-label summary from `/outputs/pseudo_labeled/` If `ffmpeg` is unavailable, emit input and augmented MEDIA from the same local run copy and still provide augmentation + auto-labeling summaries. For demo runs (no user video provided), explicitly state that input came from `nvidia/video-data-augmentation-demo`. ## Supporting files Use these canonical locations: - Workflows: `assets/configs/osmo/*.yaml` - Runtime scripts: `scripts/*.sh`, `scripts/*.py` - Flow walkthroughs: `references/flows/*.md` - Setup and triage: `references/setup.md`, `references/troubleshooting.md` - Images and endpoint policy: `references/container-images.md`, `references/nim/README.md` - Cookbook tuning: `assets/cookbooks/TUNING_GUIDE.md`

Related skills

Media

Nano Banana Pro

Generate/edit images with Nano Banana Pro (Gemini 3 Pro Image). Use for image create/modify requests incl. edits. Supports text-to-image + i…

Media

Youtube Watcher

Fetch and read transcripts from YouTube videos. Use when you need to summarize a video, answer questions about its content, or extract infor…

Media

Atxp

Access ATXP paid API tools for web search, AI image generation, music creation, video generation, X/Twitter search, email, and agent account…

Media

Youtube

YouTube Data API integration with managed OAuth. Search videos, manage playlists, access channel data, and interact with comments. Use this …