Imported from ruihangdu/CS372_Final_Project (experiment/data_prep/AGENTS.md). Install upstream with npx skills add ruihangdu/CS372_Final_Project --skill data_prep. Copyright stays with the author.

data_prep/ — Agent Instructions

Parent: experiment/AGENTS.md

Purpose

All data ingestion, transformation, and preparation. Converts raw MMOB data (video clips, audio files, CSV metadata) into the formats needed by evaluation scripts: muxed videos, sampled frames, Whisper transcripts, and manifests.

Pipeline Steps

Run in this order. Each step depends on the previous:

• id_mapping.py — Read MMOB CSV + scan filesystem, resolve hyphen/underscore ID issue, produce clip_manifest.csv
• mux_video_audio.py — Mux video + audio into single mp4 files → muxed_clips/ (1,902 files, ~1.2GB)
• sample_frames.py — Sample 1fps frames from 25fps source → sampled_frames/ (1,396 clip dirs, ~768MB)
• resample_frames.py — Resample at 5fps + 2fps for pilot 200 → sampled_frames_5fps/, sampled_frames_2fps/
• transcribe_audio.py — Whisper transcription, pilot 200 clips → transcripts/ (375 JSON files, ~2.8MB)
• transcribe_pilot.py — Alternative pilot transcription script (same output, different entry point)
• build_pilot_sample.py — Stratified sampling of 200 clips → pilot_sample.csv
• M1.1: Full-dataset Whisper transcription (Coding Agent) — Extended to all ~1,900 clips
  • All 1,902 clips transcribed successfully
  • Results at transcripts/ directory
  • transcript_manifest.csv updated to cover all clips
• M1.4: Transcript quality spot-check (Senior Data Scientist Agent) — PASS
  • Sampled 20 clips, verified JSON structure
  • 100% success rate, plausible word counts, no gibberish
  • Report: data_prep/transcript_quality_report.md
• build_sample_500.py — Stratified 500-clip sample for full experiment
  • Strata: v0_a0: 200, v1_a1: 125, v1_a0: 100, v0_a1: 75 (seed=42)
  • Output: selected_500_ids.csv (clip_id column only; use as --subset arg)
  • Run once: python data_prep/build_sample_500.py

Pending Extensions

• 5fps frames for ALL clips: resample_frames.py currently only processes pilot 200. Extend it (or write a new script) to generate 5fps frames for all ~1,875 clips with CSV annotations. Phase C evaluation requires 50 frames per clip at 5fps
  • NOTE: May not be needed if Phase C uses native video+audio input instead of frames+transcript

Data Locations (all on /scratch only)

None of these directories exist in ~/CS372/experiment/ — they are too large for git.

Manifests (CSV files — git-tracked)

ID Mapping Gotcha

This is the single most common source of bugs in this project.

The MMOB CSV (mob.csv) uses hyphens in YouTube video IDs (e.g., -1LDk-jhDFg). The filesystem converts hyphens to underscores in filenames (e.g., _1LDk_jhDFg_0.mp4). The clip_manifest.csv uses filesystem-style IDs (underscores) as clip_id — this is the canonical format used by all downstream scripts.

id_mapping.py handles the resolution:

• csv_to_fs: maps CSV videoID to filesystem videoID prefix
• fs_to_csv: reverse mapping
• The rsplit("_", 1) pattern splits clip_id into (fs_video_id, timestamp_index)

If you write new scripts that join against mob.csv, you must convert IDs. Use the mapping in id_mapping.py or the clip_manifest.csv which has already resolved everything.

Whisper Configuration

Known Issues

• Whisper misses song lyrics. In the pilot, at least one clip ("smoke weed everyday") had its lyrics completely missed by Whisper. This caused Gemini Condition A to rate the clip 0-4, while Condition B (with native audio) rated it 13-17. Music-heavy clips with embedded lyrics are a systematic blind spot
• ~40.5% of pilot clips have speech detected (81/200). The has_speech field is determined by no_speech_probability < 0.5 AND word_count > 0
• No-speech clips are rated ~0.4 tiers lower than speech clips in Condition A (information loss from missing audio context)

Transcript JSON Schema

Each file at transcripts/{clip_id}.json:

{
  "clip_id": "04BCprIjKKk_0",
  "text": "full transcribed text here",
  "segments": [
    {"start": 0.0, "end": 2.5, "text": "segment text"}
  ],
  "language": "en",
  "language_probability": 0.95,
  "duration": 10.0,
  "no_speech_probability": 0.12
}

Script Details

id_mapping.py

• Input: mob.csv, video filesystem, audio filesystem
• Output: clip_manifest.csv
• Hardcoded paths: Yes — all point to /scratch
• Idempotent: Yes (overwrites output)

mux_video_audio.py

• Input: clip_manifest.csv
• Output: muxed_clips/{clip_id}.mp4
• Dependencies: imageio_ffmpeg (provides ffmpeg binary)
• Skip logic: Skips if output file already exists and is non-empty
• Timeout: 120s per clip

sample_frames.py

• Input: 25fps source frames at mob_video/MOB_RUN/frames/video/{benign|malicious}/{clip_id}/
• Output: sampled_frames/{clip_id}/frame_00.jpg through frame_09.jpg
• Sampling: Every 25th frame from 25fps source = 1fps (indices 0, 25, 50, ..., 225)
• Also produces: frame_manifest.csv

resample_frames.py

• Input: Same 25fps source frames
• Output: sampled_frames_5fps/{clip_id}/frame_000.jpg through frame_049.jpg (and similarly for 2fps)
• Sampling: 5fps = every 5th frame, 2fps = every 12th frame
• LIMITATION: Only processes clips in pilot_sample.csv (200 clips). Must be extended for Phase C

transcribe_audio.py / transcribe_pilot.py

• Input: .wav files from mob_audio/all_audio/{videoID}/
• Output: transcripts/{clip_id}.json, transcript_manifest.csv
• transcribe_audio.py: Has PILOT_LIMIT = 200. Change to 0 or remove the limit for full-dataset run
• transcribe_pilot.py: Reads pilot_sample.csv, only transcribes pilot clips. More targeted but same output format
• Audio path lookup: Must search subdirectories because audio is organized by videoID, not clip_id

build_pilot_sample.py

• Input: clip_manifest.csv, frame_manifest.csv
• Output: pilot_sample.csv
• Stratification: (video_class, audio_class) -> v0_a0: 80, v1_a1: 50, v1_a0: 40, v0_a1: 30
• Seed: 42 (deterministic)
• Eligibility filter: Must have CSV annotations AND audio AND sampled frames

build_sample_500.py

• Input: clip_manifest.csv, frame_manifest.csv
• Output: selected_500_ids.csv (clip_id column only)
• Stratification: (video_class, audio_class) -> v0_a0: 200, v1_a1: 125, v1_a0: 100, v0_a1: 75 (total: 500)
• Seed: 42 (deterministic)
• Eligibility filter: Must have CSV annotations AND audio AND sampled frames
• Usage: python data_prep/build_sample_500.py
• Pass the output to Phase C runners via --subset data_prep/selected_500_ids.csv