vss-deploy-detection-tracking-3d

Imported from nvidia/skills (skills/vss-deploy-detection-tracking-3d/SKILL.md) via skills.sh. Install upstream with npx skills add nvidia/skills --skill vss-deploy-detection-tracking-3d. Copyright stays with the author (Apache-2.0).

Purpose

Deploy and operate the RTVI-CV-3D microservice as MV3DT (MODE=mv3dt) — per-camera DeepStream perception plus BEV Fusion over multiple calibrated cameras — on the bundled sample dataset, custom videos, or live RTSP, without the full warehouse agent / LLM / VLM stack.

Instructions

Work top-to-bottom: answer the routing questions (Q0–Q3) under Routing, then follow the reference for the chosen path. Detailed step-by-step procedures live in references/ (deploy, calibration chain, camera configuration, verification, teardown, troubleshooting).

Examples

• Enable multi-camera tracking on the sample dataset.
• Deploy RTVI-CV-3D on my videos here: <path/to/videos>.
• Run MV3DT on RTSP streams after calibration.

VSS Deploy Detection & Tracking — 3D (RTVI-CV-3D / MV3DT)

Bring up the RTVI-CV-3D microservice as the MV3DT stack (MODE=mv3dt) from the warehouse blueprint: per-camera DeepStream perception (vss-rtvi-cv-mv3dt) + BEV Fusion (vss-rtvi-cv-bev-fusion) + mosquitto MQTT bus + broker + VST sensor stack — without the agent / LLM / VLM stack that comes with the full warehouse blueprint.

The actual compose machinery lives in deploy/docker/industry-profiles/warehouse-operations/warehouse-mv3dt-app/. This skill drives the env overrides, calibration chain, and verification.

Routing

Ask the user at most four questions, then dispatch.

Q0 — Profile size (overlays or not)

Default to extended unless the user explicitly asks for minimal. Extended deploys ELK + vss-video-analytics-api-mv3dt + vss-kibana-init-mv3dt + vss-import-calibration-output-mv3dt on top of MV3DT core — these are what the VST video wall needs to render bounding-box overlays. Without them, the video wall works but shows raw streams without overlays.

Note on selective ELK: there's no "minimal + ELK only" middle path in the current compose. Every ${MINIMAL_PROFILE:+_extended}-gated service comes up together (ES, Logstash, Kibana, video-analytics-api, kibana-init, import-calibration). bash's :+ parameter expansion produces the _extended suffix when MINIMAL_PROFILE is set; extended switches the gating string back to plain bp_wh_kafka_mv3dt which the active compose profile already matches. Either you accept the full extended bundle or you stay minimal.

Q1 — Data source

Ask this unless the source is explicit in the user's first message. A bare request like "deploy rtvi-cv-3d" routes to this MV3DT skill (MODE=mv3dt), but does not imply sample.

• sample — the bundled 4-camera synthetic dataset (warehouse-4cams-20mx20m-synthetic). Calibration ships in-tree; no AMC run needed.
• videos — the user has local video files (any *.mp4 named after their cameras). Standalone AMC (auto_calib profile) will run if calibration is missing.
• rtsp — the user has live RTSP URLs. Calibration via VIOS-driven AMC; final deploy also needs a Sensor Info File (camera_info.json) with those RTSP URLs.

Q2 — Calibration coverage (skip for sample)

For videos and rtsp, check whether calibration is already on disk at the mount path the perception container expects:

DATASET="${SAMPLE_VIDEO_DATASET:?}"          # the user's dataset slug; see Q3
CAL_DIR="${VSS_APPS_DIR}/industry-profiles/warehouse-operations/warehouse-mv3dt-app/calibration/sample-data/${DATASET}"

# Look for ANY of: calibration.json, plus camInfo/*.yml or *.yaml with either
# 'cam_*' or 'Camera*' naming (the shipped sample uses Camera*.yml, AMC may
# produce cam_*.yaml — broaden accordingly)
test -f "${CAL_DIR}/calibration.json" \
  && ls "${CAL_DIR}/camInfo/"*.{yml,yaml} 2>/dev/null

If the user supplied a calibration path themselves, validate that path instead — don't recompute. See configure-cameras.md for camera-name normalization and authoritative camera-count discovery (parses calibration.json).

Q3 — Detector + dataset slug (only when Q2 triggers AMC)

• resnet (default, fast) or transformer (slower, better under occlusion) — passed to the AMC /v1/calibrate/<id> API at Step B (see vss-generate-video-calibration/SKILL.md:48-62).
• A short kebab-case dataset slug used as SAMPLE_VIDEO_DATASET (e.g. customer-aisle-4cams). This drives the calibration mount path and gets persisted in .env.

Routing table

Every path converges on references/verify-and-view.md once up -d completes. references/troubleshooting.md and references/teardown.md are linked but off the happy path.

Disambiguation rule. In this skill, "RTVI-CV-3D" means the MV3DT microservice deployment and uses MODE=mv3dt. Route to ../vss-deploy-profile/references/warehouse.md only when the user asks for the full warehouse blueprint, Sparse4D, MODE=3d, or warehouse-3d-app. This skill is for MV3DT only without the agent stack / LLM / VLM.

Prerequisites

1. Repo path

Locate video-search-and-summarization/ on disk. All compose commands run from <repo>/deploy/docker/. If unknown, ask the user.

2. NGC CLI + key

$NGC_CLI_API_KEY must be set and must have access to nvidia/vss-core/* images. See vss-deploy-profile/references/ngc.md for setup if missing.

If the user previously ran ngc config set but $NGC_CLI_API_KEY isn't exported in this shell, the key is already on disk:

NGC_CLI_API_KEY=$(awk -F'= ' '/^apikey/{print $2}' ~/.ngc/config 2>/dev/null)
test -n "${NGC_CLI_API_KEY}" && echo "key sourced from ~/.ngc/config"

Make sure the key value also lands in industry-profiles/warehouse-operations/.env:164 (NGC_CLI_API_KEY=...) — compose only reads it from there at up time, not from your shell env.

3. HARDWARE_PROFILE slug

The public MV3DT supported stream counts are listed in the Warehouse Quickstart Guide under "MV3DT Vision AI Profile Supported Deployment Options." Use the matching HARDWARE_PROFILE slug below.

Pick from nvidia-smi --query-gpu=name --format=csv,noheader:

If the user's GPU is not listed here, check industry-profiles/warehouse-operations/.env for available HARDWARE_PROFILE values, then confirm the matching profile exists in blueprint-configurator/blueprint_config.yml before using it. Do not infer a stream count from the slug alone.

The per-GPU MV3DT cap is enforced at deploy time. vss-configurator-mv3dt computes final_stream_count = min(NUM_STREAMS, max_streams_supported) and applies a keep_count file-management op against ${VSS_DATA_DIR}/videos/${SAMPLE_VIDEO_DATASET}/ so only final_stream_count .mp4 files remain (sorted lexicographically, last N kept). If your GPU's MV3DT supported stream count (above table) is below your camera count, perception / mdx-raw / mdx-bev run with the supported stream count. Either pick a GPU with a higher supported stream count or surface the cap explicitly to the user so they're aware which streams will be processed.

4. App data on disk

VSS_DATA_DIR must point at the extracted vss-warehouse-app-data directory (separate from the repo). Pointing it at the repo's deploy/docker/ causes the deploy to stall: the configurator can't find the dataset, redis can't open its log file, and perception stays in Created. Verify the path before deploy.

Pre-flight check before deploy:

DATA_DIR="${VSS_DATA_DIR:?VSS_DATA_DIR not set in .env}"
DATASET="${SAMPLE_VIDEO_DATASET:-warehouse-4cams-20mx20m-synthetic}"

for sub in videos models data_log; do
  test -d "${DATA_DIR}/${sub}" || { echo "ERROR: ${DATA_DIR}/${sub} missing"; exit 1; }
done

# For sample / videos modes — videos directory must exist
test -d "${DATA_DIR}/videos/${DATASET}" \
  || { echo "ERROR: ${DATA_DIR}/videos/${DATASET} missing — wrong slug or app-data not extracted"; exit 1; }

# Sanity: video count should match calibration count.
# Some published app-data tarballs are known to ship the sample dataset with
# fewer videos than the dataset name implies — verify and source any missing
# cams separately if your GPU's mv3dt cap is high enough to use them all.
ls "${DATA_DIR}/videos/${DATASET}/"*.mp4 2>/dev/null | wc -l

# Ensure every per-service subdir under data_log/ exists. kafka / elasticsearch /
# redis / postgres and the video-analytics API upload path (`/web-api-app/files`)
# run as non-root UIDs against these bind mounts. Without write access the daemons
# or calibration/image import can fail with permission errors.
mkdir -p \
  "${DATA_DIR}/data_log/analytics_cache" \
  "${DATA_DIR}/data_log/calibration_toolkit" \
  "${DATA_DIR}/data_log/elastic/data" \
  "${DATA_DIR}/data_log/elastic/logs" \
  "${DATA_DIR}/data_log/kafka" \
  "${DATA_DIR}/data_log/redis/data" \
  "${DATA_DIR}/data_log/redis/log" \
  "${DATA_DIR}/data_log/vss_video_analytics_api"

# Grant write access to the specific container UIDs only — scoped ACLs, NOT 777 and
# NOT chown. UIDs (per data-directory.md): postgres=70, redis=999, elasticsearch / VST /
# kafka=1000. The first call covers existing files; the second sets *default* ACLs so
# files/dirs the daemons create at runtime (e.g. postgres PGDATA) inherit the access.
ACL='u:70:rwx,u:999:rwx,u:1000:rwx'
setfacl -R    -m "$ACL" "${DATA_DIR}/data_log"
setfacl -R -d -m "$ACL" "${DATA_DIR}/data_log"

Scoped ACLs, not chmod 777. This grants only the known container UIDs access — it does not make data_log world-writable, and it does not chown (which would break postgres / Elasticsearch, since they re-own their dirs on first start). Prefer this for agent-driven runs and shared hosts. The canonical ../vss-deploy-profile/references/data-directory.md documents the broad chmod -R 777 and the per-container UID table; this skill uses the scoped-ACL equivalent instead. Ask the user for confirmation before changing host permissions.

Requires a POSIX-ACL filesystem (ext4 / xfs — the default) and the acl package (setfacl). If a daemon still logs a permission error after deploy, find its UID (docker inspect <container> --format '{{.Config.User}}') and add -m u:<uid>:rwx to both calls.

If app-data isn't extracted yet: download via ngc registry resource download-version "nvidia/vss-warehouse/vss-warehouse-app-data:<version>" and tar -xvf (see references/deploy-rtvi-cv-3d-stack.md for tag discovery and full steps).

5. Pre-flight (system)

nvidia-smi, NVIDIA Docker runtime visible (docker info | grep -i runtimes), and docker run --rm --gpus all ubuntu:24.04 nvidia-smi all green. Full driver / kernel / sysctl checks live in vss-deploy-profile/references/prerequisites.md.

If any check fails, fix before continuing — don't proceed to deploy.

6. Browser reachability (cloud / corp-VPN hosts only)

If the user will view the VST video wall through a browser on a different network than the deploy host (cloud VM, corp VPN, ssh-tunnelled session), upstream firewall rules may block VST WebRTC (STUN to stun.l.google.com:19302, plus random UDP for media). See references/verify-and-view.md#browser-reachability for symptoms and workarounds. Also: some hosts block the AMC microservice's default port (TCP/8010); if the user reports the AMC UI on :5000 works but its data calls fail, retry with a different VSS_AUTO_CALIBRATION_PORT.

Troubleshooting

When any deploy, calibration, or verification step fails, stop and classify the failure before retrying. The quick checks below cover the most common MV3DT errors; use references/troubleshooting.md for full diagnostic commands and fixes, ../vss-generate-video-calibration/SKILL.md for AMC workflow failures, and ../vss-deploy-profile/references/warehouse-debug.md for broader warehouse-stack issues.

Before destructive recovery (docker compose down -v, clearing data_log, deleting VST sensor state, or changing host ACLs), explain the impact and get user confirmation. Capture the failing command, relevant .env values, docker compose ps, and the last container logs before making state-reset changes.

How it fits together

SKILL.md (this file — Q0/Q1/Q2/Q3 routing)
  └─ if cal missing ─> calibration-workflow.md
  │                     └─ chains to vss-generate-video-calibration (deploy + drive API)
  │                     └─ fetches /v1/result/{project_id}/mv3dt_result?result_type=amc (plus vggt when refinement is enabled)
  │                     └─ lands calibration files at warehouse-mv3dt-app/calibration/sample-data/<slug>/
  ├─> configure-cameras.md (camera-name normalization, NUM_STREAMS sync, VST sensor trim)
  └─> deploy-rtvi-cv-3d-stack.md (compose up with bp_wh_kafka_mv3dt + extended/minimal)
        └─> verify-and-view.md (FPS, fusion_ready, mdx-bev, VST video wall + WebRTC checks)

Related Skills

• vss-generate-video-calibration — the AMC skill. Owns AMC deployment, RTSP capture, calibration API, and the /v1/result/.../mv3dt_result export hook this skill consumes. calibration-workflow.md chains into it.
• vss-deploy-profile — cross-profile umbrella. Use that instead when the user wants the full warehouse blueprint (with agents / LLM / VLM), not just MV3DT.
• vss-manage-video-io-storage — VIOS / VST API skill. Useful for the VST video wall (overlay viz) and for sensor management referenced in configure-cameras.md.

The repo's authoritative warehouse-blueprint reference at ../vss-deploy-profile/references/warehouse.md covers 2D / 3D / MV3DT inside the full warehouse stack — this skill is the MV3DT-only companion that trims the agent / LLM / VLM layer.