markcut

Imported from lalalic/markcut (AGENTS.md). Install upstream with npx skills add lalalic/markcut. Copyright stays with the author.

markcut — AGENT.md

Markdown-to-video engine. Describe scenes in markdown, get a rendered video with TTS narration.

Quick Start

npm run render storyboard.md                        # render markdown → MP4 (9x16 default)
npm run preview storyboard.md --edit                # live preview with auto-reload
npm run preview storyboard.md --storyboard          # fast structure preview (skips TTI/TTV, shows prompts as placeholders)
npm test                                             # unit + integration tests
npm run typecheck                                    # TypeScript type check

Input formats

See docs/markdown-strict-descriptive.md for the complete markdown syntax reference.

Key concepts

• Scene — narrative unit with name, script (TTS), instruction (visual intent), children
• Layout — parallel (simultaneous), series (sequential), transitionSeries (sequential with transitions)
• Script → TTS → STT — script field generates edge-tts audio, whisper transcribes to VTT subtitles
• Theme — preset colors/fonts/effects: cinematic, neon, minimal, corporate
• TTL config — root-level tts.cli, tts.voice, tts.rate with per-scene tts overrides
• Subtitle — root-level VTT overlay from merged per-clip STT. Not a tree node. Supports animated caption types (typewriter, fade, bounce, glowing, etc.) via remotion-subtitle, HTML in cue text, and per-cue <Sequence> rendering for zero CPU when inactive.

Stream types

CLI

markcut render <file.json|.md> [--output path]
markcut preview <file.json|.md> [--edit] [--label] [--port 3001]

Project structure

src/
├── entry.tsx              → MarkCut + DescriptiveComposition
├── descriptive/           → Compiler, markdown parser, resolve pipeline
├── types/                 → React renderers (one per stream type)
├── schema/                → Zod stream tree schemas
├── themes/                → Theme presets + ThemeProvider
├── render/cli.mjs         → CLI entry
├── render/tts.ts          → TTS via CLI template + variable substitution
├── player/pipeline.mjs    → Bundled pipeline (server imports this)
├── player/server.mjs      → Unified server (--edit, --label, preview)
├── player/components/      → React control components (edit, label, variant bar)
│   ├── index.ts
│   ├── EditControls.tsx
│   ├── LabelControls.tsx
│   └── VariantBar.tsx
└── tests/                 → Vitest integration tests

External component contract

Register components at render time:

<MarkCut
  root={descriptiveJson}
  compose={{ components: { AnimatedHeadline, StatCounter } }}
/>

In descriptive markdown:

- component componentName:StatCounter duration:3 props:{value:42}

Components receive props as defined in the descriptive JSON. No eval, no JSX parsing — just React.createElement(registry[name], props).

## Stream Tree Architecture

### Core Concept

Everything is a **stream tree** — nested JSON nodes keyed by `type`.
The tree is validated by Zod and rendered by Remotion via recursive React components.

```json
{ "id":"root", "type":"root", "width":1080, "height":1920, "fps":30,
  "isSeries":true, "transition":"fade", "children":[...] }

Composition Rules

Duration Calculation

Computed by getDurationInSeconds() in src/utils/index.ts:

• Leaf nodes: duration = leafEnd(stream) = end ?? (start + duration) (single deterministic read; start/end/duration live on the base node)
• Series: sum of children durations minus transition overlaps
• Parallel: max of children durations
• Background children: excluded from calculation
• Include with src: treated as leaf (duration from base end)
• Sets durationInSeconds on every node as a side effect

12 Stream Types

All types share base fields from BaseShape: id, name, title, description, script, src, style, visible, isBackground, start, end, duration, startFrom, endAt, durationInSeconds.

Key Implementation Details

Folder.tsx — Series/Parallel Engine

• FolderLeaf is the recursive workhorse that renders all stream types
• For isSeries:true: uses Remotion <Series> or <TransitionSeries> (if transition set)
• For isSeries:false: renders children as parallel <Sequence> elements
• Each child's duration comes from child.durationInSeconds (set by getDurationInSeconds)
• Transition elements are interleaved between series children
• Background children are wrapped in <Loop>
• Scene nodes delegate to SceneLeaf → FolderLeaf (identity)
• Effect nodes delegate to EffectWrapper → FolderLeaf

Video.tsx — startFrom/endAt Trimming

• Uses Remotion <OffthreadVideo> for CPU-efficient video decoding
• startFrom / endAt on the base node trim the source video (in seconds)
• playbackRate is capped at 1 (Math.min(1, ...)) so source longer than timeline plays at normal speed and is truncated by the Sequence
• Critical formula: OffthreadVideo endAt = startFrom_frames + ((endAt_source - startFrom_source) * fps / playbackRate) This sets endAt beyond the actual source end when playbackRate < 1, which prevents Remotion from rendering blank frames before the Sequence ends.
• endAt without explicit value falls back to stream.durationInSeconds ?? end - start

Rendering Pipeline

1. Root.tsx reads getInputProps() → parses via Zod rootSchema
2. getDurationInSeconds() stamps durationInSeconds on every node
3. Composition.durationInFrames is set to total duration
4. MarkCut wraps everything in ComposeContext + ThemeProvider + AbsoluteFill
5. FolderLeaf recursively renders the tree

CLI Usage

node src/render/cli.mjs render <file.json> [--output path]
node src/render/cli.mjs render --template <id> --data <data.json>
node src/render/cli.mjs preview <file.json> [--edit] [--label] [--port 3001]
node src/render/cli.mjs templates

Player Server (Edit Mode)

• --edit flag starts a live-reload loop: edit JSON → player auto-refreshes
• --label flag starts a labeling UI for selecting and annotating media
• --port flag sets the server port (default 3001)
• Unified server: server.mjs serves all modes (preview, edit, label). React control components in components/.
• Persistent AI agent via pi --mode rpc (one cold start, many edits): at boot, startAgentProcess() spawns the configured agent CLI (DEFAULT_AGENT_CLI) with --mode rpc appended (the -p/--prompt flag is stripped — prompts arrive over stdin). Each /api/edit POST writes {"id","type":"prompt","message"} to stdin and resolves on the streamed agent_settled event (NOT idle-timeout guessing — pi's default interactive mode is a TUI, not a stdin REPL, so text/stdin mode does not work). Assistant text is collected from message_end events (message.role==="assistant", text content blocks). --session-id (derived from the file path) resumes conversation history from disk across restarts. Do not revert to one-shot spawn per edit — that re-incurs the full cold-start every request.

Testing

Test Framework

• Vitest with custom config in tests/vitest.config.ts
• Non-parallel (pool: "forks", fileParallelism: false)
• 600s timeout per test (integration tests render real MP4s)

Key Test Utilities (tests/utils.ts)

Test Categories (30 tests)

1. Basic Rendering — empty composition, dimension verification
2. Image + Subtitle — image rendering, inline/VTT/karaoke subtitles
3. Effects — fadeIn, bounceIn, custom keyframes
4. Map Rendering — canvas route visualization
5. Audio Rendering — audio track, non-silent verification
6. Include — nested composition via include
7. Full Feature Combination — multiple features
8. Scene Node — scene-based rendering
9. Audio STT — speech-to-text verification via whisper
10. Multiple Aspect Ratios — 16x9, 9x16, 1x1
11. Cross-Stream Types — parallel rendering, effect-wrapped children
12. Frame-Accurate — timeline consistency, frame count
13. Description Field — schema validation, render preservation
14. Scene as Folder Alias — scene/folder interchangeability
15. Video startFrom/endAt Trimming — video trimming + photo in series

Running Tests

npm test                    # all tests
npm run test:integration    # render integration tests only
npx vitest run tests/render.test.ts -t "Test Name"  # single test

Adding a new built-in component

Built-in components are always available in jsx:"..." fields without imports. To add one:

1. Create the component in src/components/<Name>.tsx — a standard React component.
2. Export from barrel — add the re-export in src/components/index.ts and add the name to BUILTIN_COMPONENT_NAMES in src/components/builtin-registry.ts.

3-Level Authoring Workflow (see SKILL.md)

Script → TTS → STT → VTT Pipeline

Every node carries an optional script field for narration/dialogue text. The pipeline:

1. Storyboard — agent writes script on any node (scene, folder, leaf) as narrative text
2. TTS — walk the tree, collect all script values with node id + timing, generate WAV audio per segment via edge-tts
3. STT — transcribe each TTS audio or concatenated audio to VTT via whisper
4. Subtitle — attach resulting VTT as a subtitle child node for final rendering

This separates concerns: script = authoring text (human/agent-friendly), subtitle = rendered output (timed VTT cues).

Multi-turn Dialogue

When script text contains multiple SpeakerName: text lines (2+), the pipeline auto-expands it into a multi-turn dialogue. Each line becomes a separate audio node with speaker field, wrapped in a series container for sequential playback.

- script "Ray: Hello everyone
Alice: Good day to you
Ray: Let's begin"

Per-speaker voices are configured via voices on root. Each value is extra CLI flags appended to the TTS template:

voices:{"Ray":"--voice en-US-GuyNeural","Alice":"--voice en-US-JennyNeural"}

The speaker name is matched against voices map and the extra flags are appended to the TTS CLI template, naturally supporting voice cloning or edge-tts features (rate, pitch, etc.). Subtitles include the speaker prefix (Ray: Hello everyone).

Dependencies

• Remotion 4.0.469 — core rendering engine
• React 19 — component model
• Zod 4 — schema validation
• @remotion/transitions — scene transitions (fade, slide, wipe, flip, clockWipe)
• @vis.gl/react-google-maps — map visualization
• immer — state management (player app)
• vitest — test runner
• TypeScript 5.6 — strict mode with noUncheckedIndexedAccess