ditherer

Imported from gyng/ditherer (AGENTS.md). Install upstream with npx skills add gyng/ditherer. Copyright stays with the author.

Ditherer — Agent Guidelines

Project Overview

Ditherer is a browser-based image/video processing tool. Users load an image, select a dithering algorithm (or other effect), adjust parameters via a control panel, and apply the filter. The app also supports video frame processing, palette extraction, and state export/import via URL.

Stack: React 19, Vite, TypeScript, Rust/WASM (color space conversions), cmdk + Radix UI for the filter picker.

Quick reference: npm run dev (dev server) · npm run build (production build to build/) · npm run test (Vitest) · npm run lint (eslint).

Architecture

Component Hierarchy (Atomic Design)

The UI follows an implicit atomic design pattern:

Tokens — Design primitives defined in CSS custom properties:

• Colors: --light-gray, --gray, --beautiful-blue, --bg-color
• Layout values are inline (no token system yet)

Atoms — Leaf control components in src/components/controls/. Each renders a single HTML control:

• Range.tsx — <input type="range"> with editable value display
• Bool.tsx — <input type="checkbox">
• Enum.tsx — <select> dropdown
• Stringly.tsx — <input type="text">
• Textly.tsx — <textarea>

Molecules — Composed controls:

• Palette.tsx — palette selector + nested atom controls for palette options
• ColorArray.tsx — color swatch grid + palette extraction UI (stateful)

Organisms — Sections of the app:

• Controls (src/components/controls/index.tsx) — dispatches to the right atom/molecule based on optionTypes.type
• ChainList (src/components/ChainList/index.tsx) — filter chain editor, presets, drag/drop reordering
• Exporter (src/components/App/Exporter.tsx) — URL/JSON state export panel
• SaveAs (src/components/SaveAs/index.tsx) — image/video file export dialog (uses MediaRecorder for video)

Page — Single page app:

• App (src/components/App/index.tsx) — top-level layout: sidebar controls + draggable canvas area

State Management

App state is managed via React Context + useReducer in src/reducers/filters.ts. Components consume state via useFilter() (src/context/useFilter.ts). No external state library.

Key state shape:

• chain — array of ChainEntry { id, displayName, filter, enabled } (max 16 entries)
• activeIndex — which chain entry is selected for editing
• selected — compat shim derived from chain[activeIndex]
• inputImage / outputImage — source and processed canvases
• video — video element for realtime filtering
• Scale, grayscale, playback, linearize, wasmAcceleration

The package runtime owns ordered chain execution and temporal-state semantics. FilterContext.tsx bridges its persistent maps (prevOutputMapRef, prevInputMapRef, emaMapRef) into that runtime and owns worker selection, app caching, dispatch, and frame-loop scheduling. The reducer is pure data shape; the context handles side effects.

Filter System

Filters are the core domain (more than 300 catalog entries). Each filter is a self-contained module in packages/ditherer-filters/src/filters/ exporting:

// Every filter exports this shape
export const optionTypes = {
  paramName: { type: RANGE, range: [0, 255], step: 1, default: 128, desc: "What this controls" },
};

export const defaults = {
  paramName: optionTypes.paramName.default,
};

const filterFunc = (input: HTMLCanvasElement, options = defaults) => HTMLCanvasElement;

export default {
  name: "FilterName",
  func: filterFunc,
  options: defaults,
  optionTypes,
  defaults,
  description: "One-line user-facing summary",
  mainThread: true, // optional — only if the filter needs the temporal pipeline (see below)
};

The optionTypes declaration drives the UI — the Controls component reads it and renders the appropriate atom/molecule for each option. This is a data-driven UI pattern: filters declare what controls they need, the framework renders them. Every option should have a desc so users get tooltips.

Control type → component mapping:

Adding a new filter: Create a new file in packages/ditherer-filters/src/filters/, define optionTypes, defaults, and the filter function, then register it in packages/ditherer-filters/src/filters/index.ts (both the import and a filterList entry with displayName/category/description). If the filter is worker-capable (mainThread is not true), it must also be present in the filterIndex registry in that same file or the browser worker path will silently skip it. The UI controls are generated automatically from optionTypes.

Temporal Pipeline

Filters can read state from previous frames via injected options:

These are populated by FilterContext and persist across calls in main-thread refs.

mainThread: true flag. Filters that read _prevOutput/_prevInput/_ema, hold module-level state across calls (ring buffers, accumulators), or use dispatch MUST declare mainThread: true on their default export. Without it the filter chain runs in a Web Worker where the temporal state and module state don't persist, and the filter silently does nothing.

FilterContext.chainNeedsMainThread checks this flag — there is no central name list. Adding a new temporal filter only requires setting the flag on its export.

Existing temporal filters: motion detect, motion heatmap, motion pixelate, long exposure, frame blend, temporal edge, temporal color cycle, phosphor decay, after-image, chronophotography, slit scan, time mosaic, freeze frame glitch, video feedback, wake turbulence, background subtraction, datamosh, e-ink (ghosting), VHS (line persistence), oscilloscope (phosphor), reaction-diffusion, cellular automata, matrix rain, the error-diffusion factory (when temporalBleed > 0), and analog static (when persistence > 0).

WASM Module

packages/ditherer-filters/src/wasm/rgba2laba/ contains a Rust crate compiled to WASM for performance-critical color space conversions (RGB to CIE Lab). Loaded via dynamic import with JS fallback if WASM fails to load.

Acceleration capability flags (noGL / noWASM)

Filters can declare on their defineFilter export when a backend fundamentally can't accelerate them. The string is the short reason shown in the inline-timing tooltip — so the UI tells you "don't ask us to port this" instead of inviting another optimise request.

export default defineFilter({
  name: "Floyd-Steinberg",
  func: /* … */,
  noGL: "error diffusion is sequential; GL is gather-only. Use Ordered for parallel dithering.",
});

When to set these:

• noGL — the algorithm has a hard sequential dependency on previous output pixels that a fragment shader can't express (error-diffusion kernels: Floyd-Steinberg, Atkinson, Jarvis, Sierra, Stucki, Burkes, etc.). Anything gather-parallel (per-pixel compute, separable blurs, coordinate remaps, threshold-matrix dither) is GL-friendly and should be ported rather than flagged.
• noWASM — the filter's hot path is dominated by Canvas2D calls (graphics primitives, fillRect, drawImage composites) that Rust/WASM can't replace without re-implementing Canvas2D, OR it's so trivial that WASM marshalling overhead dominates.

What's already covered:

• errorDiffusingFilterFactory.ts sets noGL on every kernel it produces.
• Parallel dithering (Ordered, Halftone) has no noGL and is GL-accelerated.

Don't flag a filter just because its GL port hasn't landed yet — only flag when the algorithm fundamentally can't be expressed in a fragment shader.

New filters: GL-only (requiresGL: true) by default. When the filter is gather-parallel (per-pixel compute, neighborhood reads, coordinate remaps, sampling history textures) write the GL path and skip the JS fallback. A naive JS implementation is too slow for video and just adds dead code paths to maintain. Only ship a JS fallback when the algorithm is genuinely sequential (error diffusion) or trivially cheap, or when the filter must run in environments where WebGL2 is unavailable. The dispatcher renders a "WebGL2 required" stub for requiresGL filters on unsupported hardware, so users see why it didn't run.

WebGL shader validation is a release gate. npm run test:gl launches a real Chromium WebGL2 context, forces GPU acceleration across the complete filter registry, and validates every path that compiles or draws. It also requires requiresGL filters to issue an actual draw, initializes temporal GPU state, and rejects transparent or opaque-black output, preventing silent passthrough and black-frame regressions. Run it after adding or changing GLSL; unit/jsdom tests cannot compile shaders.

Raymarching and ray tracing

The raymarching family treats the loaded image as height, silhouette, material, emission, or environment data. Its registered filters are Heightfield Raymarch, Silhouette Extrusion, Voxel Landscape, Glass Surface, Relief Reflections, Volumetric Light, SDF Melt, Fractal Portal, and Path-Traced Diorama.

The extended worlds/materials suite adds Luminance Caverns, Black Hole Lens, Thin-Film Iridescence, Subsurface Wax, Cone-Traced AO, Chromatic Prism Tracer, Portal Hall, Image Fossil, Volumetric Cloud Sculpture, and Raymarched Maze. The maze uses a connected binary-tree layout and derives a guaranteed route to the exit in shader; keep that connectivity invariant if its wall logic changes.

• Keep shader loops compile-time bounded and use uniform-controlled early exits.
• Use packages/ditherer-filters/src/utils/glSinglePass.ts only for generic source-backed full-screen passes; filters continue to own their shader, controls, and uniforms.
• Filters that evolve procedural samples set temporal and autoAnimate. Path-Traced Diorama accumulates through its per-entry _prevOutput history.
• These filters are WebGL2-only and must pass npm run test:gl; the browser gate is the authoritative shader compile, enum-branch, and real-draw validation.

Filter Chains

Filters compose into chains (max 16 entries). The chain is the unit of work — the package runtime runs each enabled entry sequentially, feeding one output into the next, while FilterContext supplies app-level caching and scheduling. State is serialized to URL hash and localStorage so users can share or save chains.

Curated chain presets live in src/components/ChainList/presets.ts (CHAIN_PRESETS). To add a preset, append an entry referencing existing filter displayNames — no code change needed.

When auditing or pruning presets, run npm run report:presets first. The report flags exact duplicate preset signatures and surfaces high-similarity preset pairs so cleanup decisions are based on the current preset data rather than eyeballing the list.

Directory Structure

packages/ditherer-filters/
  src/
    filters/          # Filter modules and canonical registry
    gl/               # Shared WebGL2 execution and texture pools
    palettes/         # Palette definitions and registry
    constants/        # Control and color-space tokens used by the engine
    utils/            # Core color, canvas, sampling, and buffer helpers
    wasm/rgba2laba/   # Rust/WASM RGB-to-Lab acceleration
    workers/          # Worker executor, RPC client, and wire types
    index.ts          # Public catalog/runtime entry
src/
  components/
    App/              # App organism, Exporter, SaveAs export dialog
    ChainList/        # Filter chain editor + presets
    controls/         # Atom and molecule UI controls
    FilterCombobox.tsx # Searchable filter picker (cmdk + Radix popover)
  context/            # FilterContext — app bridge, scheduling, and dispatch
  reducers/           # App state reducer
  utils/              # App-only media, sharing, audio-viz, and policy helpers
  styles/             # Global styles
docs/plan/            # Numbered implementation plans (010 = filter audit, etc.)

Best Practices

Test-Driven Development

Write tests when they protect behavior that could realistically regress. A good test fails for the bug or missing capability, passes only when the intended behavior works, and survives harmless refactors.

Write tests first for:

• Bug fixes — reproduce the user-visible failure before fixing it.
• Pure functions — color math, buffer operations, equalize, quantize, and deterministic state generation. Cover boundaries and meaningful branches.
• Filter logic — use small known buffers or signal patterns and assert the important output property or pixel values.
• Reducers/state — test action → externally observable state transitions.
• Contracts — verify registries resolve, serialized state round-trips, and backends agree where parity is promised.

Do not add a test merely to mirror edited literals or prove that a line of configuration changed. Avoid assertions tied to private helper calls, exact internal names, incidental ordering, or a fixed list of hand-picked records unless that exact shape is a product contract. For data-only edits such as a preset swapping one registered filter for another, prefer the existing registry, signature, and end-to-end validation. Add a new test only when the edit reveals an uncovered invariant worth protecting.

Testing Pyramid

Keep the suite weighted toward fast, deterministic tests:

1. Unit tests (most) — pure utilities, reducers, option normalization, deterministic temporal state, and small-buffer filter math.
2. Integration and contract tests (fewer) — filter registration, chain execution, serialization, worker/main-thread parity, palette passes, and CPU/GL/WASM backend agreement.
3. Browser end-to-end tests (few) — WebGL shader compilation, real Canvas behavior, media playback/export, and critical user workflows that jsdom cannot represent.
4. Visual and performance checks (targeted) — reference images, signal metrics, and benchmarks for effects where appearance or throughput is the contract. Use tolerances and document the renderer/hardware.

Before adding an expensive browser test, ask whether a lower layer can protect the same behavior. Before adding a unit test, ask whether it asserts a durable outcome rather than restating the implementation. Use the smallest layer that would have caught the regression.

Use Vitest for unit/integration tests and Playwright for real-browser behavior. Tests live in test/ mirroring src/ structure.

Coverage follows the same pyramid. npm run test:coverage measures the Vitest layer; browser suites emit source-mapped V8 coverage for WebGL, WASM, workers, media, and React integration; npm run coverage:merge combines both. CI uses npm run test:coverage:gate. Do not remove a difficult module from the denominator to raise the percentage—route it through the browser layer or extract deterministic decision logic. A “does not throw” sweep is useful smoke coverage, but it does not replace branch, failure-recovery, or output-contract tests.

Code Style

• No premature abstraction. Three similar lines > one clever helper.
• Filters are self-contained. Don't create cross-filter dependencies. Shared logic goes in utils/.
• Data-driven UI. Declare controls via optionTypes, don't create custom UI per filter.
• Mutate buffers in place for performance (image processing operates on Uint8ClampedArray). Clone canvases when you need a clean copy.
• Keep filter functions pure where possible — take a canvas, return a canvas. Side effects (async dispatch) are the exception, not the rule.

Component Guidelines

• Atoms are stateless function components. They receive a value and an onChange-style callback.
• Molecules compose atoms and may hold local UI state (e.g., ColorArray's extract mode toggle).
• The Controls dispatcher (controls/index.jsx) is a switch on type — keep it flat, don't nest logic.
• CSS Modules for component styles. Global styles only in src/styles/.
• For draggable floating windows that use position: fixed + transform, compute drag offsets from the element's live getBoundingClientRect() at mouse-down time. Using cached position refs can cause a visible snap on the first drag after mount/remount.
• Reuse the shared chrome tokens for option/section headers. controls/styles.module.css exports .optionGroup, .optionGroupLegend, and .subsectionHeader. Compose them at every section header instead of redefining font-size / font-weight / text-transform locally — that's how header styles drift across panels. If you need different spacing or color, create a local class that composes the canonical token.

Performance

• Image processing is CPU-bound. Keep filter hot loops tight — avoid allocations, use typed arrays.
• WASM for expensive color math (Lab distance). JS fallback must exist.
• Memoize expensive conversions (e.g., wasmRgba2labaMemo).
• requestAnimationFrame for video frame processing — don't block the main thread.
• WebGL2 benchmarks default to swiftshader (software renderer) — GL numbers are slower than real GPU hardware. When benchmarking GL vs WASM, use _webglAcceleration: false to isolate the WASM path. The GL fast path is still correct and will outperform WASM on a real GPU; swiftshader results do not indicate a regression for end users.
• A real GPU is reachable — PLAYWRIGHT_GPU=1 PLAYWRIGHT_ANGLE=gl routes Playwright through Mesa's d3d12 driver to the actual adapter, headless, with nothing needed in your shell. Required for any question a software rasterizer can't answer (register pressure, occupancy, "fast enough for video"), because swiftshader and llvmpipe model none of it. Print UNMASKED_RENDERER_WEBGL and read it before trusting a GL number — limits differ enough to mislead you (MAX_FRAGMENT_UNIFORM_VECTORS is 4096 on swiftshader, 1024 on an RTX 3080). Wall-clock GL timing measures the readback, not the shader; use EXT_disjoint_timer_query_webgl2, report spread, and reproduce before believing a result. See docs/gpu-benchmarking.md for the method, the traps, and a worked example of getting it wrong.
• WASM load timing: wasmIsLoaded() returns false until the async WASM init resolves. Benchmarks that run immediately after page load will see JS fallback numbers for all WASM-accelerated filters. Always wait for WASM before measuring (see bench.ts for the pattern).

Plans

Implementation plans live in docs/plan/. Numbered chronologically:

• 001 — modernization (complete)
• 002 — gamma-correct pipeline
• 003 — wide-gamut color
• 004 — js → ts migration (complete)
• 005 — realtime perf
• 006 — filter list organization
• 007 — filter chaining
• 008 — algorithm optimization
• 009 — temporal filters (pipeline shipped, several filters built)
• 010 — filter audit (descriptions, blue noise, 11 new temporal filters, presets)
• 011 — export dialog (SaveAs)

When making non-trivial changes, write a plan first under docs/plan/NNN-name.md and reference it from the commit.