media_slurp

Imported from JQLSpec/media_slurp (AGENTS.md). Install upstream with npx skills add JQLSpec/media_slurp. Copyright stays with the author.

Repository Guidelines

Project Structure & Module Organization

• media_transcoder.py holds the watcher loop, configuration loader, FFmpeg command builders, and watchdog event handlers. Treat it as the canonical source for business logic and architecture.
• media_transcoder_cfg.json defines root paths (base, ingest, Camera_Originals, Encoded/H265, Encoded/Proxy), encoder presets, file extensions, and processing thresholds. Copy it to ~/.media_transcoder_cfg.json for user-specific overrides.
• check_video_metadata.sh and set_camera_metadata.sh (documented in SHELL_SCRIPTS.md) live at repo root and provide ffprobe/ffmpeg helpers for camera metadata hygiene.
• Documentation (README.md, SHELL_SCRIPTS.md, this AGENTS.md file) and dependency manifest (requirements.txt) are also in the root directory; there are no nested packages or tests yet.

Build, Test, and Development Commands

# Install Python dependencies (watchdog)
pip3 install -r requirements.txt

# Run the transcoder interactively (creates ingest/output folders automatically)
python3 media_transcoder.py

# Audit metadata before ingest (ensures camera/date tags are present)
./check_video_metadata.sh Ingest/*.mp4

Coding Style & Naming Conventions

• Indentation: 4 spaces throughout the Python codebase; no tabs observed.
• File naming: Snake_case for scripts (media_transcoder.py, check_video_metadata.sh), uppercase for supporting docs (README.md). Config files use lowercase with underscores.
• Function/variable naming: Python functions and locals use snake_case, classes use CapWords (e.g., Config, VideoTranscoder). Shell scripts prefer uppercase env vars and mixed-case echo labels.
• Linting: No automated linters are configured. Follow PEP 8 manually and keep shell scripts POSIX-compliant. If adding tooling (e.g., ruff, shellcheck), document the commands in this section.

Testing Guidelines

• Framework: No automated tests ship with the repository. Validation relies on manual runs of media_transcoder.py plus the metadata helper scripts.
• Test files: When adding tests, mirror the root layout (e.g., tests/test_transcoder.py) and keep fixtures under tests/fixtures/.
• Running tests: Not applicable yet; document pytest/unittest commands once introduced.
• Coverage: No coverage targets are defined. For now, sanity-check by transcoding a short sample clip and verifying outputs in Encoded/H265 and Encoded/Proxy.

Commit & Pull Request Guidelines

• Commit format: The existing history favors short imperative statements (e.g., Initial things, Update). Prefer <scope>: <summary> if you introduce more complex work, but keep subjects ≤50 chars and bodies wrapped at 72 chars.
• PR process: Reference the feature or fix in the PR description, summarize hardware/codec changes, and note any config migrations. Include manual test evidence (sample command output) before requesting review.
• Branch naming: No enforced scheme is documented. Suggested pattern: feature/<short-topic> or fix/<issue-id> so contributors can infer purpose quickly.

Repository Tour

🎯 What This Repository Does

Media Transcoder automates ingest-to-delivery processing for camera footage by watching an ingest folder, stabilizing files, then launching parallel FFmpeg jobs that emit visually lossless H.265 masters and Final Cut Pro–friendly H.264 proxies.

Key responsibilities:

• Monitor ingest paths and defer work until files finish copying
• Detect camera metadata to build YYYY-MM-DD/Camera folder hierarchies
• Select the best available hardware encoder (VideoToolbox, NVENC, VAAPI, or libx265/libx264 fallback)

🏗️ Architecture Overview

System Context

[Camera card dump / Watch folder]
           ↓
   [media_transcoder.py]
           ↓                      ↘
   Encoded/H265 & Proxy      Camera_Originals (archive)
           ↓
      Final Cut Pro / NLEs

Key Components

• Config (class in media_transcoder.py): Loads JSON config (user override → project default), resolves platform-specific base paths, and probes hardware acceleration by shelling out to sysctl, nvidia-smi, or /dev/dri checks.
• VideoTranscoder: Orchestrates ffprobe scans, metadata extraction, FFmpeg command synthesis, progress parsing, and post-processing (size reporting, original relocation).
• VideoFileHandler & Observer loop: Watchdog-based filesystem event handler that debounces file writes, queues stable files, and invokes the transcoder; main() wires the observer, processes existing backlog, and keeps the loop alive.

Data Flow

1. Watchdog notices a new supported extension inside Ingest/ and tracks its last-write timestamp.
2. Once stability_timeout elapses without changes, VideoTranscoder.transcode_file() probes duration, frame count, recording date, and camera type (ffprobe tags, filename heuristics).
3. Date/camera subfolders under Encoded/H265 and Encoded/Proxy are created; two FFmpeg commands (H.265 + proxy) run on dedicated threads with hardware-aware flags.
4. Progress lines are parsed from FFmpeg stderr for live frame/fps/ETA output. When both encodes succeed, the original file is moved to Camera_Originals/ and size savings are reported.

📁 Project Structure [Partial Directory Tree]

media_slurp/
├── media_transcoder.py         # Main watcher + transcoder + CLI entry point
├── media_transcoder_cfg.json   # Default paths, encoder presets, stability tuning
├── README.md                   # End-user setup, hardware notes, services
├── SHELL_SCRIPTS.md            # Metadata helper documentation
├── check_video_metadata.sh     # ffprobe audit helper for camera/date tags
├── set_camera_metadata.sh      # ffmpeg helper that injects tags in-place
├── requirements.txt            # Python deps (watchdog 4.0.0)
└── AGENTS.md                   # This AGENTS.md file (guidelines + tour)

Key Files to Know

🔧 Technology Stack

Core Technologies

• Language: Python 3.7+ – chosen for cross-platform filesystem access, subprocess control, and available threading.
• Frameworks/Libraries: watchdog==4.0.0 powers filesystem monitoring; threading, subprocess, and pathlib handle concurrency and process orchestration.
• Media Tooling: FFmpeg/ffprobe (external binaries) provide probing, hardware acceleration, and encoding; VideoToolbox (macOS), NVENC (NVIDIA), VAAPI (Intel/AMD), and libx265/libx264 serve as encoder backends.
• Shell Utilities: launchd/systemd examples enable background service deployment.

Key Libraries

• Watchdog – Observer + event handler abstractions for cross-platform directory watching.
• FFmpeg – Actual encoder/decoder invoked from Python (handles hevc/h264 video, AAC audio, metadata copying).
• ffprobe – Supplies duration, frame count, metadata (creation_time, make/model) for routing logic.

Development Tools

• pip – Installs the lone Python dependency.
• brew/apt – Referenced in README for FFmpeg installation.
• launchctl/systemd – Optional service managers when running unattended.

🌐 External Dependencies

Required Services

• FFmpeg + ffprobe – Must include hardware acceleration support relevant to the host (VideoToolbox on macOS, NVENC/VAAPI on Linux). Absence falls back to software encoders with slower throughput.
• GPU drivers / hardware interfaces – nvidia-smi for CUDA availability; /dev/dri/renderD128 for VAAPI; Apple Silicon/Intel Quick Sync on macOS detected via sysctl.

Optional Integrations

• launchd/systemd units – Provide auto-start behavior; scripts live outside this repo but README contains templates.

🔄 Common Workflows

Watch-folder Transcoding

1. Copy raw footage into Ingest/ under the configured base directory.
2. media_transcoder.py ensures directories exist, processes any backlog, then watches for new files.
3. After stability checks, the file is transcoded twice in parallel, outputs appear in Encoded/H265/<date>/<camera>/ and Encoded/Proxy/<date>/<camera>/, and the source moves to Camera_Originals/.
4. Import outputs into Final Cut Pro or other NLEs using the date/camera hierarchy.

Code path: main() → Config.ensure_directories() → VideoFileHandler.process_pending_files() → VideoTranscoder.transcode_file().

Metadata Repair Loop

1. Run ./check_video_metadata.sh Camera_Originals/*.mp4 to find files organized under Unknown.
2. For each offender, execute ./set_camera_metadata.sh <file> <make> <model> to apply QuickTime + standard tags (optionally inferring creation_time from filenames).
3. Re-ingest fixed files by moving them back into Ingest/; the watcher reprocesses them with correct folder routing.

Code path: Shell scripts (check_video_metadata.sh, set_camera_metadata.sh) → media_transcoder.py metadata parsing helpers (_get_camera_type, _get_recording_date).

📈 Performance & Scale

• Parallelism: H.265 and proxy jobs run on separate threads, so total job time roughly matches the slower encoder. Hardware accelerators dramatically improve throughput (3–10× real-time depending on GPU/CPU).
• Stability heuristics: stability_required_checks and stability_check_interval prevent processing half-copied files; adjust for slower storage if needed.
• Timeouts: encode_timeout defaults to two hours; long-form footage on software encoders may need higher limits.

Monitoring

• Live console output includes platform, selected encoders, per-job frame/fps/ETA, and size comparisons.
• Extend logging by honoring the logging section in media_transcoder_cfg.json (log file name + level).

🚨 Things to Be Careful About

🔒 Security & Safety

• File moves are destructive: Successful jobs move originals into Camera_Originals/. Ensure that directory is backed up if footage must be preserved elsewhere.
• Metadata scripts overwrite files: set_camera_metadata.sh deletes the original after writing a tagged copy. Always keep backups or run on duplicates first.
• External binaries: FFmpeg commands are assembled from config and metadata; validate user-provided overrides to avoid injecting unsafe parameters.

Data Handling

• Sensitive footage paths live in config files; avoid committing user-specific overrides in ~/.media_transcoder_cfg.json.
• Directory permissions (especially /Users/Shared/media or /home/media) must allow the watcher to create subdirectories and move files.

Update to last commit: d6b8bf5

Last updated: 2026-02-15