About

Imported from nvidia/skills (skills/tao-analyze-gaps-vlm-bcq/SKILL.md) via skills.sh. Install upstream with npx skills add nvidia/skills --skill tao-analyze-gaps-vlm-bcq. Copyright stays with the author (Apache-2.0).

VLM Binary Classification Gap Analysis

Name: tao-analyze-gaps-vlm-bcq
Author: nvidia

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report.

Purpose

After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

Usage

Invoke the vlm_bcq action inside the TAO Toolkit data services container with Hydra-style key=value overrides:

gap_analysis vlm_bcq \
  predictions_json=/path/to/results.json \
  results_dir=/path/to/output/gaps

Include videos_dir when video_id values in the predictions are relative paths:

gap_analysis vlm_bcq \
  predictions_json=/path/to/results.json \
  results_dir=/path/to/output/gaps \
  videos_dir=/path/to/videos/root

After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.

Inputs

predictions_json: Path to predictions JSON file. Must be a JSON array where each item has video_id, response, and gt fields. response and gt are parsed with word-boundary matching — 'yes' or 'no' anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
videos_dir (optional): Base directory for resolving relative video_id paths. If omitted, video_id values are used as absolute paths.

Predictions JSON format:

[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

Outputs

kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields: video_id (absolute path), error_type (FP or FN), question, ground_truth, response.
kpi_gaps_report.txt: Human-readable table with total FP/FN counts.

If no gaps are found, no files are written and a message is logged.

Key Parameters

Parameter	Required	Description
predictions_json	Yes	Path to predictions JSON file
results_dir	Yes	Output directory; created if it does not exist
videos_dir	No	Base directory for resolving relative `video_id` paths

Error Patterns

Error	Cause	Fix
`FileNotFoundError`	`predictions_json` does not exist	Check the path
`ValueError: must be a JSON array`	Predictions file is not a list	Wrap predictions in `[...]`
`ValueError: missing 'gt'/'response'/'video_id'`	A prediction item is missing a required field	Inspect and fix the predictions JSON
Samples silently skipped	`response` or `gt` contains both or neither 'yes'/'no'	Check logs for warnings; inspect those samples

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/nvidia-skills-tao-analyze-gaps-vlm-bcq/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

nvidia-skills-tao-analyze-gaps-vlm-bcq.ocm.jsonjson

{
  "ocm": "1",
  "id": "nvidia-skills-tao-analyze-gaps-vlm-bcq",
  "kind": "skill",
  "name": "tao-analyze-gaps-vlm-bcq",
  "description": "Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to \"analyze VLM BCQ gaps\", \"extract VLM false positives and false negatives\", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.",
  "publisher": "nvidia",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "gap-analysis",
      "rcca",
      "vlm",
      "evaluation",
      "false-positive",
      "false-negative",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to \"analyze VLM BCQ gaps\", \"extract VLM false positives and false negatives\", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/nvidia/skills",
      "path": "skills/tao-analyze-gaps-vlm-bcq/SKILL.md",
      "ref": "HEAD",
      "url": "https://www.skills.sh/nvidia/skills/tao-analyze-gaps-vlm-bcq",
      "key": "nvidia/skills/skills/tao-analyze-gaps-vlm-bcq/SKILL.md"
    },
    "compatibility": "Requires docker + nvidia-container-toolkit.",
    "allowed_tools": [
      "Read",
      "Bash"
    ],
    "license": "Apache-2.0"
  },
  "instructions": "# VLM Binary Classification Gap Analysis\n\n> **Standalone install?** If this session was not initialized by the TAO skill bank plugin, run the `tao-setup` skill first (host preflight, credentials, cross-skill discovery).\n\nReads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report.\n\n## Purpose\n\nAfter running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false ",
  "cost": {
    "context_tokens": 786
  }
}

Fetch it by URL: GET /api/v1/registry/nvidia-skills-tao-analyze-gaps-vlm-bcq/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.