About

Imported from nvidia/skills (skills/paidf-augmentation/SKILL.md). Install upstream with npx skills add nvidia/skills --skill paidf-augmentation. Copyright stays with the author (Apache-2.0).

PAIDF Augmentation Pipeline Skill

Name: paidf-augmentation
Author: nvidia

Unified pipeline for augmenting camera data through NVIDIA generative AI models with automated captioning, generation, and quality evaluation. BYOM (bring-your-own-model): every model is reached over a remote HTTP endpoint described by one entry in the config's endpoints: list; adding a model is usually a config change, not code.

Purpose

Use this skill to drive the PAIDF augmentation pipeline end to end:

Select the right model — Cosmos Transfer 2.5 (transform a video), Cosmos Predict 2.5 (generate/extend video), image-edit (edit an image), or image-to-video (animate a first frame: Cosmos3 or Veo 3.1).
Author and validate YAML configs against the PipelineConfig Pydantic schema.
Configure captioning (VLM, LLM, deterministic VLM-template, text, or file) and evaluators (hallucination check, attribute verification, VLM verification).
Launch and run inference inside the paidf-augmentation:1.1.0 Docker container (remote-API only — no local model weights).

Use this skill when running inference, authoring or editing configs, debugging validation or runtime errors, adding data samples, configuring captioning, tuning generation parameters, registering BYOM endpoints/adapters, or setting up evaluators. Trigger keywords: augmentation, cosmos transfer, cosmos predict, image edit, image-to-video, veo, image attribute augmentation, defect image generation, captioning, attribute verification, config validation.

Do not use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.

Prerequisites

Requirement	Detail
Docker	`docker --version`. The image is remote-API only — it bundles no Cosmos/torch weights, so plain remote inference needs no GPU and no `HF_TOKEN`.
NVIDIA GPU (conditional)	Only for the `data_processing.alignment` post-processor (cupy) and H.264 decode (evaluators, `data_processing.transcode`). See Limitations.
Endpoint URLs	One reachable URL per role the config uses: the model role (`video_transfer`/`video_predict`/`image_edit`/`image2video`) plus `vlm`/`llm` for captioning and evaluation. Defaults are local Qwen vLLM servers (`Qwen/Qwen3.6-27B-FP8` on `vlm`, `Qwen/Qwen2.5-14B-Instruct` on `llm`). If the user has none running, ask for URLs.
API keys (conditional)	Only for endpoints requiring auth. Passed by env var named in each endpoint's `api_key_env` — never hardcoded in YAML. Common: `VLM_API_KEY`, `LLM_API_KEY`, `VEO_API_KEY`, `BUILD_NVIDIA_API_KEY`. Local endpoints need none.
Input media	A video (transfer/predict) or image (edit/image2video) reachable by `multistorageclient` — local path, `s3://`, `gs://`, `az://`, or HTTP.

Inputs

Resolve each value in this precedence order: state file → explicit prompt arguments → agent context → user prompt. Ask the user only for what remains unresolved.

Input	Required	Description
`config_path`	Yes	Path to the pipeline YAML, e.g. `configs/cookbook/video-data-augmentation/config_video_transfer_CT25_nim.yaml`. If absent, pick a starting config from Supported Models and confirm with the user.
`input_media`	Yes	Source video/image → `data[].inputs.rgb`. Overridable at run time via `data.0.inputs.rgb=...`.
`output_paths`	Yes	`data[].output.{video,caption,metadata}`; `evaluation` optional.
`model_name`	Yes	`augmentation.model.name` — an endpoint `id`, a role, or a known model name. Free-form string, not an enum.
`endpoint_urls`	Yes	One `endpoints[]` entry per role in use.
`api_key_env`	If auth	Env-var name per endpoint; the value comes from the environment.
`target_attributes`	No	`captioning.llm.variables` (e.g. `weather_condition`, `lighting_condition`).
`generation_params`	No	`augmentation.parameters` — pass-through; only set knobs are sent.
`seed`	No	Under `augmentation.parameters`; `null` = random, re-rolled on retry.

BYOM model: endpoints, adapters, roles

The pipeline never embeds an SDK per model. Instead:

endpoints: is a list. Each entry has role, url, model (the wire model string), an optional id (only to disambiguate 2+ endpoints sharing a role), an optional adapter (API contract; defaults from the role), api_key_env, and timeout.
Roles: vlm, llm (captioning + evaluators), image_edit, video_transfer (Cosmos Transfer), video_predict (Cosmos Predict), image2video (Cosmos3 / Veo).
Adapters (API contracts): openai.chat.completions, openai.images.edits, openai.video.sync, openai.video.async, nim, passthrough. The same model can be served over different contracts by changing only the endpoint's adapter field.
Model selection: augmentation.model.name resolves to an endpoint by id, else by role, else by the model-name→role map (image-edit→image_edit, cosmos-transfer2.5→video_transfer, cosmos-predict→video_predict, cosmos3-image2video→image2video).

Supported Models

When the user hasn't specified a model, choose from their input type and goal:

Input Type → Goal	`model.name`	Role / default adapter	Input → Output
Video — change scene attributes (weather, lighting, style)	`cosmos-transfer2.5`	`video_transfer` / `nim`	Video (+ controls) → Video
Video + text — extend or predict continuation	`cosmos-predict`	`video_predict` / `nim`	Video+Text → Video
Text only — generate video from scratch	`cosmos-predict` (`inference_type: text2world`)	`video_predict` / `nim`	Text → Video
Image — edit specific attributes	`image-edit`	`image_edit` / `nim` (or `openai.chat.completions`, `openai.images.edits`)	Image → Image
Image — animate a first frame	`cosmos3-image2video` (or your Veo endpoint id)	`image2video` / `openai.video.sync` (Veo: `openai.video.async`)	Image + prompt → Video

Key rule: video in + scene-attribute change → Cosmos Transfer. Generate new video from text/image/video conditioning → Cosmos Predict. Single image edit → image edit. Still image → moving clip → image-to-video.

All models run via remote HTTP through one BaseExecutor; there is no local torchrun and no executor_type field.

Usage

Step 1: Launch the Docker Container

Set PAIDF_IMAGE_ID to the immutable sha256: image ID recorded from the trusted local build (or supplied in trusted release metadata). The image ID is build- and architecture-specific, so this repository cannot provide one universal value. Verify that the mutable convenience tag still resolves to the expected ID, then run the ID directly:

set -e

PAIDF_IMAGE_ID="sha256:<expected-image-id>"
test "$(docker image inspect --format '{{.Id}}' paidf-augmentation:1.1.0)" = "$PAIDF_IMAGE_ID"
docker network inspect paidf >/dev/null 2>&1 || \
  docker network create paidf

docker run -it --rm \
  --network paidf \
  -v "$(pwd)/modules:/workspace/modules" \
  -v "$(pwd)/configs:/workspace/configs" \
  -v "$(pwd)/data:/workspace/data" \
  --entrypoint /bin/bash \
  "$PAIDF_IMAGE_ID"

Do not derive PAIDF_IMAGE_ID from the tag and immediately trust it; compare the tag against the digest recorded when the image was built or published. If a registry release provides a signed manifest, verify that signature before pulling and use its name:tag@sha256:<manifest-digest> reference instead.

Networking: augmentation only makes outbound requests, so it needs no -p/--publish ports. Keep the shared paidf bridge shown above for remote endpoints. For another model container, attach it to the same bridge and use its container name in the endpoint URL. Run host-local models in a container on that bridge, or use a remote endpoint; do not grant the augmentation container access to the host network.
API keys: prefer a platform secrets manager that injects the required environment variables. Otherwise, export only the required keys and forward their names with -e VAR_NAME; never mount or load a broad credential file.
No GPU needed for remote inference — add --gpus for data_processing.alignment and any H.264 decode; pick a GPU not shared with a busy model server. Container runs as uid 10000; ensure data/ is writable (or --user "$(id -u):$(id -g)").

Security: Host networking is prohibited for this workflow, especially when API keys are present. Review pipeline-operations.md.

Step 2: Run the Pipeline (Inside the Container)

uv run --no-sync modules/cli.py --config configs/<config_file>.yaml

# With OmegaConf CLI overrides (dot-list syntax)
uv run --no-sync modules/cli.py --config configs/cookbook/video-data-augmentation/config_video_transfer_CT25_nim.yaml \
  data.0.inputs.rgb=/workspace/data/input.mp4 \
  augmentation.parameters.seed=42

Environment variables: keys resolve as the api_key_env var → the role's default env var. If api_key_env names an unset var, resolution falls back to the role default; leave it off for unauthenticated endpoints. LOG_LEVEL sets logging.

Configuration Schema

Configs are validated against PipelineConfig (modules/aug_utils/schema/) and have seven top-level sections: data, endpoints (a list), pipeline, captioning, augmentation, data_processing, and evaluators. Full per-section YAML is in configuration-schema.md; runtime flow and common editing tasks are in pipeline-operations.md.

Examples

Configs live under configs/cookbook/<use-case>/. See the cookbook index for the folder layout.

Use case	Config(s)
Video scene-attribute transfer (CT2.5, `nim`)	`config_video_transfer_CT25_nim.yaml`
Image → video	`config_image2video_cosmos3.yaml` (VLM→LLM) · `config_image2video_cosmos3_vlm_template.yaml` (VLM→template) · `config_image2video_veo31.yaml` (Veo 3.1, async)
Image Attribute Augmentation	`config_image_edit_attribute_{chat_api,images_api,nim}.yaml` · `…_gemma_llm.yaml` (hosted-Gemma LLM swap)
Defect Image Generation + MI alignment	`config_image_edit_defect_{chat_api,images_api}.yaml`
Batch config generation	`workflow_example.yaml` · `attribute_distribution_1000_v1.yaml`
Smart-space seed image / event video	`config_seed_image_gen_cosmos3_super_t2i_smart_spaces.yaml` · `config_event_video_gen_cosmos3_smart_spaces.yaml`

Per-config captioning / evaluator / adapter details are in config-decision-tree.md.

Troubleshooting

Run all inference and schema validation inside the Docker container for a consistent environment. For config-validation errors, runtime/endpoint errors, and typical per-stage timings, see troubleshooting.md.

Limitations

Remote inference only. All models run behind remote HTTP endpoints; no local weights, no torchrun, no executor_type, no Gradio executor.
GPU for alignment and H.264 decode. Remote inference needs no GPU. A CUDA GPU is required by data_processing.alignment (cupy) and by anything decoding H.264 — the evaluators and data_processing.transcode — because the image ships only the hardware h264_cuvid decoder (software AVC decode is off for licensing). VP9 decodes in software. Video output is VP9-only.
Inference only. This pipeline augments and generates media — it does not train or fine-tune models.
Auth varies by endpoint. Hosted endpoints (e.g. Veo) need a key via api_key_env; local endpoints (e.g. vLLM) need none.

Reference files

configuration-schema.md — full per-section YAML for every config section.
config-decision-tree.md — which config to start from, model/captioning selection, alignment override rules.
pipeline-operations.md — pipeline flow, worked example, common tasks, storage, security notes.
captioning-strategy-guide.md — all 6 captioning modes with complete YAML.
evaluator-setup-guide.md — hallucination tuning, attribute verification, MCQ wiring.
troubleshooting.md — validation/runtime errors and per-stage timings.
image-attribute-augmentation.md — Image Attribute Augmentation image-edit workflow and dataset packaging.
event-video-gen.md — smart-space image-to-video event generation.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/nvidia-skills-paidf-augmentation/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

nvidia-skills-paidf-augmentation.ocm.jsonjson

{
  "ocm": "1",
  "id": "nvidia-skills-paidf-augmentation",
  "kind": "skill",
  "name": "paidf-augmentation",
  "description": "Use when authoring or validating PAIDF augmentation YAML configs, or running remote Cosmos Transfer/Predict, image-edit, or image-to-video inference.",
  "publisher": "nvidia",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "physical-ai",
      "augmentation",
      "cosmos",
      "image-edit",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Use when authoring or validating PAIDF augmentation YAML configs, or running remote Cosmos Transfer/Predict, image-edit, or image-to-video inference."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/nvidia/skills",
      "path": "skills/paidf-augmentation/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/nvidia/skills/blob/HEAD/skills/paidf-augmentation/SKILL.md",
      "key": "nvidia/skills/skills/paidf-augmentation/SKILL.md"
    },
    "license": "Apache-2.0"
  },
  "instructions": "# PAIDF Augmentation Pipeline Skill\n\nUnified pipeline for augmenting camera data through NVIDIA generative AI models with automated captioning, generation, and quality evaluation. **BYOM (bring-your-own-model):** every model is reached over a remote HTTP endpoint described by one entry in the config's `endpoints:` list; adding a model is usually a config change, not code.\n\n## Purpose\n\nUse this skill to drive the PAIDF augmentation pipeline end to end:\n\n- **Select the right model** — Cosmos Transfer 2.5 (transform a video), Cosmos Predict 2.5 (generate/extend video), image-edit (edit an image),",
  "cost": {
    "context_tokens": 3240
  }
}

Fetch it by URL: GET /api/v1/registry/nvidia-skills-paidf-augmentation/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.