glowing-waffle

Imported from Luen/glowing-waffle (AGENTS.md). Install upstream with npx skills add Luen/glowing-waffle. Copyright stays with the author.

AGENTS.md

Agent-facing notes for Glowing Waffle: a fully client-side, on-device text-to-speech app. Flutter UI + Kokoro 82M for the voice, with two engine implementations — kokoro-js on web and sherpa-onnx on native. There is no server; all inference happens on the user's machine.

README.md is for humans (platforms, setup). This file is the extra context an agent needs: commands, platform splits, traps that have already bitten us, and the house rules for commits, PRs and issues.

Commands

flutter pub get                                   # deps

# Always run all three before committing; CI (.github/workflows/ci.yml) runs them.
flutter analyze
flutter test
flutter build web --release

# Web-only code paths are compiled out of VM tests (kIsWeb is false, the
# dart:js_interop branch is not loaded). Run them in a browser:
flutter test --platform chrome

# JS syntax check (web/ is excluded from flutter analyze)
node --check web/tts_bridge.js

End-to-end smoke tests use the same models and configs as the app. CI runs all of them:

scripts/download_kokoro.sh                     # stages Kokoro (~360 MB) into assets/tts/
scripts/download_kokoro.sh --kitten            # stages Kitten nano (~25 MB) into assets/tts_kitten/
python3 -m venv .venv && . .venv/bin/activate
pip install -r scripts/requirements.txt
python scripts/smoke_tts.py --out native-smoke.wav
python scripts/smoke_tts.py --engine kitten --out kitten-smoke.wav
cd scripts/web_smoke && npm ci && node smoke.mjs ../../web-smoke.wav

Run it locally:

flutter run -d chrome                          # web: no model download, Kokoro loads in-browser
flutter run -d macos                           # native: downloads the selected model on first run (~350 MB Kokoro, ~25 MB Kitten)
flutter build web --release --output=build/web-preview
python3 -m http.server 8099 -d build/web-preview   # build/web needs an http origin; file:// will not work

Where things live

Platform split

Two conditional imports decide everything:

• lib/tts/tts_engine_impl.dart → tts_engine_web.dart or tts_engine_mobile.dart
• lib/screens/home_screen.dart → home_screen_web.dart or home_screen_io.dart

So: web has one tab (Text); native adds the Web Reader tab (webview). kIsWeb is false in VM tests, so web-only UI is only exercised by flutter test --platform chrome or a real browser.

TtsEngine (lib/tts/tts_engine.dart) is an abstract class whose optional hooks have default no-op bodies. Platform engines implements it rather than extends it, so adding a method to TtsEngine breaks the build until tts_engine_mobile.dart and tts_engine_web.dart are updated too (tts_engine_stub.dart extends, so it inherits the defaults). When you add a hook, add it to the applied-on-Play list in app.dart and to the fake engine in test/widget_test.dart.

The web bridge

web/tts_bridge.js exposes window.glowingWaffleTts and is called from Dart through dart:js_interop extension types in lib/tts/tts_engine_web.dart (and lib/tts/audio_saver_web.dart, which uses the same public GET to get at the bridge). The JS object must expose exactly the names declared there.

Rules that are not obvious from reading the code:

1. Bump ?v in web/index.html whenever you change the bridge. The file is a non-hashed static asset, so browsers keep serving a stale copy while main.dart.js updates. The symptom is a hard failure such as NoSuchMethodError: method not found: 'setVoice' (q.setVoice is not a function) in the UI — that is exactly how #12 broke locally. The bridge tag carries a comment saying the same thing.
2. dtype must match device. Quantized Kokoro (q8) on the WebGPU execution provider deterministically emits corrupted, unintelligible audio that is easily misheard as "the model is speaking a foreign language"; WebGPU needs fp32, WASM stays on q8 (#13). Check loadKokoro() before touching device/dtype.
3. window.glowingWaffleTts only exists once the module has run, so the Dart getter can be null — the engine implementations already guard with ?..
4. setBackend('auto' | 'gpu' | 'cpu'), setVoice and setSpeed are pushed before every synthesis. Keep the "setting the same value is a no-op" guard: otherwise pressing Play would reset an already-loaded engine.
5. The bridge persists two things in localStorage (settings, and the last auto engine decision). An explicit backend choice deliberately does not touch the auto decision, so overriding never contaminates Auto.
6. splitSentences() in web/tts_bridge.js and splitIntoChunks() in lib/tts/text_chunker.dart must stay in sync: both engines speak the same text in the same sentence-sized chunks (300 chars), so the two implementations are deliberately identical, down to the terminator-only fallback. Change them together and update test/text_chunker_test.dart.

Models, voices and assets

• Web — kokoro-js pinned to 1.2.1 (CDN URL in the bridge, scripts/web_smoke/package.json for the smoke test) plus onnx-community/Kokoro-82M-v1.0-ONNX. Downloads happen in the browser and are cached by the browser.
• Native — sherpa-onnx with one of two models, chosen in the Model dropdown (native only):
  • Kokoro 82M (default): model.onnx 325 MB, voices.bin 27 MB, lexicon-us-en.txt, tokens.txt, espeak-ng-data/, 20 English voices.
  • KittenTTS nano 15M: model.fp16.onnx 24 MB, voices.bin, tokens.txt, espeak-ng-data/, 8 voices, no lexicon (espeak-ng phonemizes). Neither is bundled: lib/tts/model_store.dart downloads the selected model's archive on first run into <support>/tts_models/<name>, verifies the pinned sha256, extracts only the files TtsModel.files lists, and deletes the archive. The OS does not purge the support directory, unlike the temp directory this used to copy into on every cold start. lib/tts/tts_models.dart is the single source for the archives, pins, file lists and voices; --dart-define=KOKORO_MODEL_URL=… / KOKORO_MODEL_SHA256=… (and KITTEN_MODEL_URL / KITTEN_MODEL_SHA256) point a build at another archive (a mirror, a smaller quantized variant); an empty …_SHA256 skips the check. Update the pin when the URL changes.
• Model switching is native-only and has to reach the engine. The web engine keeps using Kokoro through kokoro-js and implements setModel as a no-op rather than faking a second engine, so PlaybackControls(showModelPicker:) is only passed true by the native screens. Readiness, downloadModel() and the voice list all refer to TtsAppState.model, and app.dart _applyModel pushes a change to the engine when the picker moves — without that, the readiness check still asks about the old model.
• Bundling is opt-in — declaring assets: [assets/tts/] (Kokoro, staged by scripts/download_kokoro.sh) or assets: [assets/tts_kitten/] (Kitten, --kitten) in pubspec.yaml makes the app import that copy and never download; leave either out of pubspec.yaml or a web build picks up hundreds of MB of model files. The same staged files are what the smoke tests load.
• Runtime download needs platform permissions: Android's INTERNET (android/app/src/main/AndroidManifest.xml — the main manifest, not just the debug one) and macOS's com.apple.security.network.client in both macos/Runner/*.entitlements. The WebView's own traffic is not affected by the macOS entitlement; Dart's HttpClient is.
• iOS backs up the support directory to iCloud, so downloaded models are included in a device backup today; excluding them needs NSURLIsExcludedFromBackupKey, see the open issue.
• Voices are per model: TtsModel.voices in lib/tts/tts_models.dart (Kokoro's 20 ids are in sherpa speaker-id order — list index == native sid — and kokoro-js uses the same ids on purpose; Kitten's 8 names are in voices.bin order). TtsAppState.voice resolves to the remembered pick for the selected model or that model's default, so switching can never leave an id the engine cannot use.
• Keep scripts/smoke_tts.py (ENGINES), tts_engine_mobile.dart and the bridge's kokoro-js call in sync. The smoke tests are the only thing that catches drift between them.
• Third-party licences live in THIRD_PARTY_NOTICES.md; add a row there when a model, native library or bundled asset is added. Note that sherpa-onnx compiles espeak-ng (GPL-3.0-or-later) into the native library, so native builds contain GPL-3.0 code (#32); espeak-ng's data is downloaded with the model too.

Verifying audio — do not skip this

Bad TTS output is audible, not visible, and it is very easy to fool yourself by checking that a call returned without throwing. Never call an audio bug fixed (or diagnosed) from a successful API call.

Capture the samples and transcribe them:

1. In the browser, stub AudioBufferSourceNode.prototype.start to capture this.buffer's channel data (the bridge plays through Web Audio).
2. Resample 24 kHz → 16 kHz.
3. Transcribe with @huggingface/transformers + onnx-community/whisper-tiny.en.

For "The quick brown fox jumps over the lazy dog." the broken #13 config transcribed as [speaking in foreign language]; the fixed one transcribed exactly. Also confirm the result is deterministic (two runs of the same config identical) before comparing two configs — otherwise you are chasing noise.

For native, scripts/smoke_tts.py writes a WAV and CI uploads it as the smoke-audio artifact. That covers the config; to cover the engine itself (download, setModel, _writeWav, just_audio), write a throwaway macOS integration test that drives TtsEngineMobile, copy the tts_*.wav out of getTemporaryDirectory() while playback runs (the engine deletes it when playback completes), and transcribe that. That is how the missing temp directory on macOS was caught (#26).

The model download has two moving parts that fail silently if you only check that a future completed: the pinned sha256 and the tar layout (a top-level directory to strip, entries the engine does not need). Check both against the real archive, not only a fixture:

# 1. Serve the real archive out of process, so the peak memory you measure is
#    the store's, and run a *compiled* harness: `dart run` compiles in-process
#    and adds its own ~55 MB to every peak.
python3 -m http.server 8129 -d <archive dir> &
dart compile exe <throwaway harness using ModelStore> -o /tmp/model_check
/usr/bin/time -l /tmp/model_check http://127.0.0.1:8129/<archive>.tar.bz2 /tmp/out kokoro
# Kokoro (published archive): 359 files / 377,710,739 bytes in 29 s, 382 MiB peak.
# Kitten nano: 358 files / 41,805,804 bytes in 2.5 s, 54 MiB peak.

# 2. Feed what it extracted to sherpa, the same config the app uses:
cp <extracted>/{model.onnx,tokens.txt,voices.bin,lexicon-us-en.txt} assets/tts/
python3 -c "import shutil; shutil.make_archive('assets/tts/espeak-ng-data','zip','<extracted>','espeak-ng-data')"
python scripts/smoke_tts.py --out /tmp/store-smoke.wav
# Kitten instead: assets/tts_kitten/ (model.fp16.onnx, no lexicon) and
python scripts/smoke_tts.py --engine kitten --out /tmp/kitten-store-smoke.wav

A bad extraction shows up as a shorter/empty file (the archive decoder hands the callback entries with no content unless storeData: true), so check sizes, not just that files exist.

Saved WAVs are the same audio as playback — both paths share synthesizeChunk() / one sherpa-onnx generate call — so a corrupt save means a corrupt playback path, not the exporter. test/audio_export_test.dart covers the header and PCM conversion; it does not cover audible quality.

Testing

• Tests live in test/widget_test.dart (app/UI behaviour, including the model and voice pickers), test/model_store_test.dart (the model download, for both models, against a fixture archive served from a local HTTP server; it is @TestOn('vm') because it binds a real HttpServer), test/text_chunker_test.dart (the sentence splitter) and test/audio_export_test.dart (WAV encoding, file names). Widget tests inject a fake engine through GlowingWaffleApp(engine: …) and a fake saver through GlowingWaffleApp(audioSaver: …); those are the intended seams for exercising "settings reach the engine" and "what would be written to disk", and new TtsEngine members must be added to that fake.
• Never let a widget test hit the real filesystem: always pass audioSaver.
• Test behaviour, not implementation: UI → state, and state → engine call.
• Write platform-specific assertions with skip: kIsWeb / skip: !kIsWeb so one file passes both on the VM and in Chrome; the Chrome run is what proves the real web screen (for example, that the web home screen shows a control).
• Anything that touches dart:js_interop needs flutter test --platform chrome.

House rules

• One PR per issue, always branched off main. Branch names: fix/…, feat/…, docs/…, chore/…, ci/….
• Commit messages and PR bodies explain why, and are backed by evidence. Match the existing history (git log): imperative subject, then the problem, then a Verified: block with the exact commands and observed results. Use gh (already authenticated) for issues and PRs.
• Keep PRs small and single-purpose. If a change needs a model/interface refactor and a feature, land the plumbing first and fold the rest into the issue.
• Be honest in the PR. List what you deliberately left out under "Not in this PR" and link the issue that covers it. Say when a check was run against an earlier revision instead of implying it covered the final one.
• Update README.md when user-facing behaviour, platforms or setup steps change (this file is for agents; the README is for people).
• Never commit secrets or model weights. Signing material stays in CI secrets (#11); *.jks, *.p8, *.p12, *.key, .env* are git-ignored.
• Never fake an engine. The old "Kitten" web path actually loaded Kokoro and had to be renamed (#3). If a configuration is not actually supported on a platform, hide the option.
• Nothing is written to a user-visible location unless the user asks for it. Saving is opt-in per press; playback keeps using the temp file it deletes.
• Every engine in this app renders 24 kHz mono (kKokoroSampleRate). If a model with a different rate is ever added (#6), exported WAV headers and the web chunk joins must follow the model, not this constant.
• Check flutter analyze is clean and that generated/ignored directories are untouched; web/, android/, ios/ and macos/ are excluded from analysis.

When you find something else — file an issue

Work on this repo tends to surface things that are out of scope for the change in hand: a broken edge case, a wrong assumption, dead code, a missing test, a dependency or model that misbehaves. Do not leave that knowledge in a chat message, a code comment, or a passing remark — file an issue immediately, then carry on with (or defer) the original task.

Check for duplicates first, then create it:

gh issue list --state open
gh label list
gh issue create --title "<concise, specific>" --body-file /tmp/issue.md --label bug

File an issue for:

• any error, crash or wrong output you hit while working, even if it is unrelated to your current task;
• a bug you can see but cannot fix in the current change;
• an improvement, UX gap or performance problem you notice;
• anything that needs a closer look or refactoring (duplicated logic, unclear interface, missing test, stale docs, dead code);
• work you deliberately defer from a PR;
• a discovery about a dependency, model, device or platform (for example a device/dtype combination that produces bad output).

Write it so someone without your context can act:

• Observed — exact error text, command, platform/device/browser, versions;
• Reproduce — the minimal steps or script that shows it;
• Evidence — transcription output, log line, screenshot path, a link to the CI run;
• Ruled out — what you already checked, so nobody repeats it;
• Where — the files involved, and the smallest acceptable fix if you have an opinion;
• cross-link the PR or issue you were working on when you found it.

Then reference it from the PR ("Not in this PR: #NN") so the two stay connected.

Open work (Oct 2026)

• #8 Verify Android, iOS and macOS builds on real toolchains (macOS CI job, iOS minimum version for sherpa_onnx_ios).
• #11 Store signing for release builds (secrets required).
• #14 Allow saving the generated audio to a file — PR #19 predates the model and engine refactors and now conflicts in 10 files; it needs a rebase.
• #20 iOS includes the downloaded models in iCloud device backups; excluding them needs NSURLIsExcludedFromBackupKey.
• #21 Model extraction holds the largest file in memory (the Kokoro tar peaks at ~325 MB).
• #23 TTS settings are not persisted — voice, speed, model and backend reset on every restart. Worth a shared_preferences pass.
• #24 scripts/download_kokoro.sh stages Kitten too now; rename it.
• #25 Evaluate the newer kitten-nano-en-v0_8 (and micro/mini) checkpoints.
• #27 _playGenerated waits on just_audio's state stream with no timeout, so a missing completion leaves the UI on "Playing".
• #29 IndexTTS is not a drop-in engine (needs a second runtime, GB weights and a restrictive licence); ZipVoice, via the sherpa-onnx we already ship, is the on-device voice-cloning candidate.
• #31 Kokoro multi-lang-v1_1 (103 speakers) needs a decision first: web and native deliberately share voice ids, so the two lists would have to diverge or move together.
• #32 Third-party notices now live in THIRD_PARTY_NOTICES.md, but the app has no LICENSE of its own and k2-fsa has not been asked about the linked espeak-ng (GPL-3.0).
• #33 Survey of VoiceStudio / Chatterbox / kokoro-web / IndexTTS; its actionable items are #29–#32.
• #36 The web smoke test's sharp carried three high advisories (fixed by PR #37 via an override); sprintf-js still has no patched release.