Imported from vantuh/whisperx-server (AGENTS.md). Install upstream with npx skills add vantuh/whisperx-server. Copyright stays with the author.

AGENTS.md

Project Overview

whisperx-server — a self-hosted Docker-based transcription API server powered by WhisperX. Accepts audio files via HTTP, returns transcriptions with optional speaker diarization. Supports both NVIDIA GPU (CUDA) and CPU-only modes.

Tech Stack

• Runtime: Python 3.12
• Framework: FastAPI + uvicorn
• Transcription: WhisperX (faster-whisper backend)
• Diarization: pyannote.audio (via whisperx.diarize.DiarizationPipeline)
• Container: Docker (GPU via NVIDIA passthrough, or CPU-only for testing)
• GPU: NVIDIA RTX 5080 (16 GB VRAM, Blackwell/CUDA 12.x) for production

Architecture

Client (curl/script)
  → POST /transcribe (multipart file upload + params)
  → FastAPI server (uvicorn, single worker)
    → Returns job_id immediately
    → Job enters sequential queue (single worker thread)
    → Worker processes jobs one at a time:
      → lazy-load models (if not already in memory)
      → transcribe (faster-whisper)
      → align (forced alignment)
      → diarize (pyannote, optional)
    → Updates job status at each stage
  → GET /jobs (list all jobs with statuses)
  → GET /jobs/{job_id} (poll for progress + result)
  → GET /jobs/{job_id}/txt (plain text grouped by speaker)
  ← JSON response (segments with text, timestamps, speakers)

Project Structure

whisperx-server/
├── AGENTS.md
├── README.md
├── Dockerfile               # GPU build (nvidia/cuda base)
├── Dockerfile.cpu           # CPU build (python:3.12-slim, for testing on Mac/non-GPU)
├── docker-compose.yml       # GPU deployment
├── docker-compose.cpu.yml   # CPU deployment (model defaults to "tiny")
├── requirements.txt
├── .env.example             # HF_TOKEN template
├── .gitignore
├── config.py                # Environment variables and constants
├── model_manager.py         # ModelManager — lazy load / auto-unload
├── jobs.py                  # Job dataclass + JobStore (in-memory)
├── worker.py                # Background transcription pipeline
├── routes.py                # FastAPI endpoints
├── server.py                # App creation, lifespan, entry point
├── skills/
│   └── transcribe-remote/
│       ├── SKILL.md         # AI agent skill for transcription
│       └── scripts/
│           └── whisperx-cli # Bash CLI client (curl + jq, works on macOS/Linux)
└── data/                    # SQLite DB (gitignored, Docker volume)
    └── jobs.db

Key Design Decisions

• Lazy model loading: Models load on first request (not at startup). Server starts instantly. Models auto-unload after configurable idle timeout (MODEL_IDLE_TIMEOUT, default 5 min) to free GPU memory.
• Active job tracking: ModelManager tracks _active_jobs count — won't unload models while jobs are running.
• Background job processing: POST /transcribe returns a job_id immediately. Transcription runs in a background thread. Client polls GET /jobs/{job_id} for progress.
• Job progress stages: queued → loading_model → transcribing → aligning → diarizing → done | failed
• Job TTL: Completed jobs are cleaned up after JOB_RESULT_TTL (default 1 hour).
• Sequential queue: Jobs are processed one at a time by a single worker thread. Multiple concurrent requests are queued and processed in order. This prevents GPU memory conflicts.
• Single worker: uvicorn runs with 1 worker — WhisperX is not thread-safe and GPU memory is shared.
• Temp file handling: Uploaded audio is saved to a temp file, processed, then deleted in finally block.
• Compute type: float16 on CUDA, float32 on CPU. Do not use int8 — RTX 5080 has native FP16 support.
• Batch size: 16 (16 GB VRAM is sufficient for large-v3-turbo with float16 and batch 16).
• Default model: large-v3-turbo for GPU, tiny for CPU testing.
• Diarization import: Use from whisperx.diarize import DiarizationPipeline (not whisperx.DiarizationPipeline). Constructor uses token= parameter (not use_auth_token=).

Environment Variables

API Contract

POST /transcribe

Request: multipart/form-data

Response:

{"job_id": "a1b2c3d4e5f6", "status": "queued", "progress": "Waiting in queue"}

GET /jobs

Response:

[
  {"job_id": "...", "status": "transcribing", "progress": "Transcribing audio...", "created_at": 1234567890.0, "completed_at": null},
  {"job_id": "...", "status": "queued", "progress": "Queue position: 1/2", "created_at": 1234567891.0, "completed_at": null},
  {"job_id": "...", "status": "done", "progress": "Completed", "created_at": 1234567800.0, "completed_at": 1234567850.0}
]

GET /jobs/{job_id}

Response (in progress):

{"job_id": "...", "status": "transcribing", "progress": "Transcribing audio...", "created_at": 1234567890.0}

Response (done):

{
  "job_id": "...",
  "status": "done",
  "progress": "Completed",
  "created_at": 1234567890.0,
  "completed_at": 1234567950.0,
  "result": {
    "text": "full transcription text",
    "segments": [{"start": 0.0, "end": 2.5, "text": "segment text", "speaker": "SPEAKER_00"}],
    "language": "uk"
  }
}

Response (failed):

{"job_id": "...", "status": "failed", "progress": "Failed", "error": "error message", "completed_at": 1234567950.0}

GET /jobs/{job_id}/txt

Returns plain text with consecutive segments grouped by speaker. Designed for AI agent summarization.

Response (200, text/plain):

[SPEAKER_00]: Привіт. Та, думаю, краще набереш, щоб не було цей. Зіпсо на телефону.
[SPEAKER_01]: Давай я розкажу і пояснюю. Є дві робочі групи.
[SPEAKER_00]: Вони це хто? Це команда платформи?

Response (409): Job not ready yet.

GET /health

{"status": "ok", "model": "large-v3-turbo", "model_loaded": true, "idle_timeout": 300}

Docker

GPU (production)

docker compose up --build -d

• Base image: nvidia/cuda:12.8.0-runtime-ubuntu24.04
• Requires NVIDIA Container Toolkit on host
• GPU passthrough via docker-compose.yml

CPU (testing on Mac/non-GPU)

docker compose -f docker-compose.cpu.yml up --build -d

• Base image: python:3.12-slim
• CPU-only torch (smaller image)
• Default model: tiny (fast enough for testing)
• Transcription is slow (~3x realtime) but functional

Network Access (WSL2)

WSL2 is configured with networkingMode=mirrored (C:\Users\Ivan\.wslconfig), so Docker ports are directly accessible on the Windows host IP without any port forwarding. No netsh portproxy needed.

Windows Firewall rule

Currently restricted to MacBook only (192.168.31.163):

# Allow only MacBook
New-NetFirewallRule -DisplayName "WhisperX Server" -Direction Inbound -LocalPort 8080 -Protocol TCP -Action Allow -RemoteAddress 192.168.31.163

To add more IPs:

Set-NetFirewallRule -DisplayName "WhisperX Server" -RemoteAddress 192.168.31.163,192.168.31.XXX

To disable access temporarily:

Disable-NetFirewallRule -DisplayName "WhisperX Server"

To re-enable:

Enable-NetFirewallRule -DisplayName "WhisperX Server"

To remove the rule entirely:

Remove-NetFirewallRule -DisplayName "WhisperX Server"

Constraints

• No authentication — server runs in a trusted local network only. Access restricted via Windows Firewall to specific IPs.
• SQLite persistence — job results stored in data/jobs.db (Docker volume whisperx-data). Survives container restarts.
• No WebSocket or streaming — poll-based progress via /jobs/{job_id}.
• Keep dependencies minimal — only what WhisperX and FastAPI need.
• Proper error handling: 400 for bad files, 404 for unknown jobs, 500 for pipeline errors.

Testing

# Submit a job
curl -X POST http://localhost:8080/transcribe \
  -F "file=@test.mp3" \
  -F "language=uk" \
  -F "diarize=false"

# Poll for progress
curl http://localhost:8080/jobs/<job_id>

# Health check
curl http://localhost:8080/health

# Test with diarization
curl -X POST http://localhost:8080/transcribe \
  -F "file=@test.mp3" \
  -F "language=uk" \
  -F "diarize=true"

# Test with invalid file (should return 400)
curl -X POST http://localhost:8080/transcribe \
  -F "file=@readme.md"