moss-tts.cpp

Imported from mudler/moss-tts.cpp (AGENTS.md). Install upstream with npx skills add mudler/moss-tts.cpp. Copyright stays with the author.

Maintainer's guide — moss-tts.cpp

Durable reference for humans and agents maintaining this repo. Concise on why; the README covers what users see.

AI-assisted contributions

This project follows the same policy as the sibling repos (parakeet.cpp, vibevoice.cpp) — the Linux kernel project's guidelines for AI coding assistants. Commit rules:

• No Signed-off-by from AI. Only a human submitter signs off the DCO.
• No Co-Authored-By: <AI> trailers. The human contributor owns the change.
• Use an Assisted-by: trailer to attribute AI involvement. Format: Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL] (e.g. Assisted-by: Claude:claude-opus-4-8 [Claude Code]).
• The human submitter reviews, tests, and understands every generated line.

What this project is

moss-tts.cpp is a C++17/ggml inference port of the OpenMOSS MOSS-Audio-Tokenizer (the "Cat" — Causal Audio Tokenizer with Transformer), a CNN-free, pure-Transformer RVQ neural codec: 24 kHz waveform ↔ 32 RVQ code streams at 12.5 Hz. It runs entirely on stock ggml (no fork), with no Python/ONNX/torch at inference time, and ships a flat C-API (include/moss_tts_capi.h) for dlopen/FFI/LocalAI integration.

This is the Foundation (F1 repo scaffold + F2 audio tokenizer) of a larger program: porting the whole MOSS-TTS family to ggml. The decomposition, ordered dependency-first, is F1 → F2 → V1 → V2 → V3 → V4:

FOUNDATION (this repo)  shared by every variant
  F1  repo scaffold (CMake + ggml + dr_wav + loader + audio_io + C-API/CLI)
  F2  MOSS-Audio-Tokenizer: encoder (wav→codes) + decoder (codes→wav)
        every variant decodes its audio through F2
V1  MossTTSDelay (8B)      Qwen3 backbone + 33 emb/heads + delay SM + sampling
V2  MossTTSLocal (1.7B)    depth-transformer backbone, time-sync RVQ blocks
V3  MossTTSRealtime (1.7B) hierarchical text-audio inputs + streaming
V4  MOSS-TTS-Nano (~100M)  separate upstream repo, 48 kHz

F2 comes first because every variant's audio is produced by this codec, it is the riskiest novel ggml work (ONNX-only upstream), and it can be validated fully standalone (reconstruction round-trip vs the ONNX tokenizer — no backbone). The full design + rationale is in docs/superpowers/specs/2026-06-03-foundation-audio-tokenizer-design.md.

Reference sources (single sources of truth, in order of trust)

1. OpenMOSS-Team/MOSS-Audio-Tokenizer (HF) — config.json (dims) + modeling_moss_audio_tokenizer.py (module logic). Authoritative.
2. Blaizzy/mlx-audio mlx_audio/codec/models/moss_audio_tokenizer/ — clean non-PyTorch reference for the algorithms.
3. OpenMOSS-Team/MOSS-Audio-Tokenizer-ONNX — the encoder/decoder ONNX we benchmark + parity-test against (the upstream torch-free pipeline).
4. The model's model.safetensors.index.json — authoritative tensor names.

When a numeric bug is suspected, dump the matching tensor from (1) and diff. Magnitude alone is not a reliable signal (the vibevoice lesson).

Layout

include/
  moss_tts.h            C++ API (moss::Codec: load/sample_rate/num_quantizers/encode/decode/reconstruct)
  moss_tts_capi.h       flat C-API for FFI / dlopen
src/
  moss_tts.cpp          C++ API impl (version + thin Codec pimpl over AudioTokenizer)
  moss_tts_capi.cpp     flat C-API shim
  audio_tokenizer.{hpp,cpp}  orchestrator: wires encoder towers + quantizer + decoder towers
  model_loader.{hpp,cpp}     GGUF -> config + name->ggml_tensor map; builds towers from metadata
  backend.{hpp,cpp}     ggml backend init + persistent ggml_gallocr; compute_graph_with_inputs()
  patchify.{hpp,cpp}    CNN-free down/up reshape-permute subgraph (PatchedPretransform)
  transformer.{hpp,cpp} ProjectedTransformer: RoPE MHA + LayerScale + erf-GELU FFN + sliding-window mask
  quantizer.{hpp,cpp}   ResidualLFQ: encode (argmin loop) + decode (gather+sum)
  rope.hpp              RoPE cos/sin (interleaved-pair convention)
  audio_io.{hpp,cpp}    dr_wav load/save + linear resample to 24 kHz
  common.{hpp,cpp}      logging
  ggml_extend.hpp       small ggml helpers
examples/cli/main.cpp   moss-tts-cli: info | encode | decode | reconstruct
scripts/
  convert_audio_tokenizer_to_gguf.py  safetensors -> GGUF (WNConv fusion, identity-proj drop, stage metadata)
  gen_test_fixtures.py                numpy/scipy reference dumps for the model-independent parity tests
  gen_onnx_reference.py               upstream ONNX encoder+decoder -> ref_recon.wav + ref_codes.npy (parity ref)
  requirements.txt                    converter deps + reference-only (onnxruntime, soundfile)
tests/
  test_*.cpp            ctest targets; SKIP_RETURN_CODE=77 means "skipped"
  fixtures/             committed tiny GGUF + numpy reference dumps
bench.sh                reconstruction RTF: moss-tts-cli vs ONNX on samples/*.wav
docs/
  conversion.md         authoritative GGUF tensor-naming + converter reference
  superpowers/specs/    the F1+F2 design spec
third_party/ggml        pinned submodule (no fork)
models/                 output dir for converted GGUFs (gitignored)

Memory model (carried from parakeet.cpp — do not regress)

• Subgraphs are built in a no_alloc=true ggml_context: tensor metadata exists but backing buffers do not until allocation. Inputs are then materialized via backend::compute_graph_with_inputs (src/backend.hpp), which does ggml_gallocr_alloc_graph → write each input's data → compute.
• Keep the persistent ggml_gallocr in src/backend.cpp reused across the many per-call graphs (no per-call alloc/free). Do NOT swap in ggml_backend_sched on the fast path — it re-plans the split every call and regressed CUDA in parakeet.cpp. Sched is a per-graph fallback only, used when the active GPU backend lacks an op kernel.
• Zero-copy weights: clone loader tensors by reference; never copy weights per call.
• Compute-pool sizes scale with sequence/sample length (the vibevoice lesson — hardcoded sizes silently pass on short clips and abort on long ones).

Metadata-driven towers

The encoder and decoder are 8-module sequences alternating PatchedPretransform (no weights) and Transformer stages. The loader builds the towers from the explicit moss.at.{enc,dec}.* int32 stage arrays (n_stages + the parallel per-stage arrays kind, patch, in_dim, d_model, n_heads, n_layers, d_ff, out_dim, read via get_i32_array) — no source changes per stage. The moss.at.encoder_kwargs / decoder_kwargs JSON strings are emitted for provenance only; the loader does not parse them. Encoder patch ratios 240,2,2,2 (product = 1920 = downsample); decoder mirrors 2,2,2,240.

Build

cmake -B build -DMOSS_TTS_BUILD_TESTS=ON -DMOSS_TTS_BUILD_EXAMPLES=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build --output-on-failure

CMake options

Runtime backend selection: MOSS_TTS_BACKEND=cpu|cuda|... (honored verbatim by src/backend.cpp when the named backend is registered, else falls back).

Profiling (per-stage wall-clock)

src/profiler.{hpp,cpp} provides MOSS_PROFILE("stage") RAII scopes that accumulate per-stage time into an env-gated registry. Off by default, zero overhead, inference byte-identical (the scope is a single bool check when off and never touches inference tensors). Two ways to read it:

• MOSS_TTS_PROFILE=1 ./build/bin/moss-tts-cli tts … — prints the per-stage table to stderr at the end of any tts/tts-* run.
• ./build/bin/moss-tts-cli bench <delay|local|rt|nano> --model … --codec … --tokenizer … --text "…" [--warmup 1] [--iters 3] [--seed 12345] — runs --warmup discarded then --iters measured generations and prints the table + averaged wall/audio + RTF.

Stages: embed, backbone.prefill, backbone.decode, depth (+ nested depth.head for the per-channel local heads), heads (V1 top-level head), sample, codec. The report separates one-time backbone.prefill from per-step backbone.decode via the calls/avg columns. The threshold for the head-dot parallel_for is independent (MOSS_HEAD_PARALLEL_MIN).

CAVEAT: tiny-fixture numbers are NOT representative (tiny hidden/vocab) — they exercise the instrumentation for CI/regression only; real per-stage attribution needs a real checkpoint. Complements (does not replace) the bench_*.sh end-to-end RTF scripts.

Tests at a glance

moss_add_test() sets SKIP_RETURN_CODE=77, so a test that returns 77 shows as ctest SKIP. The model-independent tests run anywhere (no checkpoint); the real-model tests SKIP until their env vars point at the 7GB checkpoint (and, for parity, an ONNX reference).

Env vars for the real-model tests (set whichever you need):

MOSS_TTS_TOKENIZER    # .gguf path (used by all three real-model tests)
MOSS_TTS_PARITY_IN    # input wav fed to the ONNX reference
MOSS_TTS_PARITY_REF   # OUT/ref_recon.wav from scripts/gen_onnx_reference.py
MOSS_TTS_BACKEND      # cpu|cuda|... (runtime backend; also used by bench.sh)

Parity-fixture methodology

Each ggml op is pinned by a numpy/scipy reference: scripts/gen_test_fixtures.py computes the expected tensors (patchify interleave, LayerNorm+GELU, RoPE tables, transformer-block forward, quantizer argmin/gather) and dumps them to tests/fixtures/; the matching test_*.cpp builds the same ggml subgraph on the same tiny weights and asserts close agreement. This is how every op was validated before the real checkpoint existed. Add a fixture when you add an op.

GGUF tensor naming + the converter

scripts/convert_audio_tokenizer_to_gguf.py turns the upstream safetensors into a single f32 GGUF. The full schema is authoritative in docs/conversion.md; the load-bearing rules:

• WNConv fusion: encoder/decoder block projections and all quantizer projections are weight-normalized 1×1 convs (weight_norm(dim=0)). The converter fuses g/v (parametrizations.weight.original0/1) into a dense (out,in) <name>.weight and keeps the stored bias. Bit-exact vs torch.
• Identity-projection drop: block projections are present only when dims differ; when in == d_model == out the projection is absent and the C++ loader treats a missing input_proj/output_proj as identity. Do not synthesize it.
• Stage-metadata arrays: per-stage block configs + patch ratios are emitted as GGUF KV int32 arrays (moss.at.{enc,dec}.{n_stages,kind,patch,in_dim,d_model,n_heads,n_layers,d_ff,out_dim}) and the loader builds the towers from these. The moss.at.encoder_kwargs / decoder_kwargs JSON strings are also emitted but for provenance only — the loader does not parse them.
• Tensor names match the upstream PyTorch hierarchy verbatim (no model. prefix). Against the real checkpoint the converter reports 0 unmapped keys.

Always run the converter with --strict to fail on unmapped source keys.

Gotchas / past-bug archaeology

From the spec's gotchas list + what the port confirmed. These will bite again.

1. No input normalization. Feed the raw float waveform in [-1,1]; only zero-pad to a multiple of 1920. No DC removal, no gain. (vibevoice normalized and broke things.)
2. erf-GELU, not tanh. Exact erf-GELU; FFN is two bias-free linears, NOT SwiGLU.
3. LayerNorm has weight AND bias (eps 1e-5). It is not RMSNorm.
4. LayerScale is a learnable per-channel vector multiplying the residual branch output, between the sublayer and the residual add. Never fold it into a linear.
5. RoPE is interleaved-pair (ggml GGML_ROPE_TYPE_NORMAL), head_dim 64, max_period 10000, applied to Q and K. Offline streaming offset = 0.
6. MHA, not GQA. Fused QKV in self_attn.in_projs.0 (3*d_model, d_model), bias-free; out_proj bias-free; SDPA scale head_dim^-0.5.
7. Sliding-window causal mask is stage-dependent: key j visible to query i iff 0 ≤ i-j < context, context = round(frame_rate_at_stage_input × 10s). Track current_frame_rate exactly through the module list (start 24000, divide per encoder downsample / multiply per decoder upsample after appending). For short clips this equals plain causal; implement the window so long audio matches the reference bit-for-bit.
8. Patchify interleave order must mirror exactly on decode. Down does (D,T)→(D,T/p,p)→permute→(D*p,T/p); up mirrors. Codes layout is ne0 = NQ, ne1 = T (frame-major). Getting the interleave wrong silently corrupts audio.
9. Quantizer L2-normalizes both the projected encoding and the codebook (eps 1e-12) before the argmin in 8-dim codebook space; decode is argmin + gather + sum only — no EMA, straight-through, or extra scaling.
10. Decoder output is the waveform directly from the final patch-240 unpatchify — no output conv, tanh, or clamp.
11. gallocr input-set ordering: with no_alloc=true, an input's buffer does not exist until ggml_gallocr_alloc_graph, so input data must be written after allocation, not at build time. Use backend::compute_graph_with_inputs (alloc → set inputs → compute); writing inline aborts.
12. Bitrate control = use the first k of 32 codebooks on encode/decode.

Known follow-ups / not yet implemented

Genuine deferrals — recorded so they are not forgotten:

• Bitrate-k control. encode/decode/CLI/C-API currently always use all 32 codebooks. The spec notes you can decode with the first k of 32 (k×0.125 kbps); exposing a --n-quantizers k control is a future enhancement. (gen_onnx_reference.py already accepts --n-quantizers on the ONNX side.)
• Codes-exact-match assertion in the parity gate. gen_onnx_reference.py emits ref_codes.npy, and the spec's parity DoD mentions asserting our encode codes match the ONNX codes above an exact-match rate, but tests/test_reconstruct_parity.cpp currently only checks reconstructed-waveform SNR. Wiring a codes comparison (and tuning the exact-match-rate threshold) is best done during the first real-model run.

V1: MossTTSDelay (8B text-to-speech)

The first generative variant. Text (+ optional reference clip for voice cloning) -> speech, decoded through the Foundation codec. Built dependency-first across T1-T13; all 33 streams (1 text + 32 audio) are produced by one Qwen3 backbone with 33 embedding tables and 33 output heads, sequenced by the "delay" staircase.

Pipeline (one synthesis)

text + instruction/language + (reference wav)
  -> de.tokenizer (Qwen3 BPE)        : text -> token ids
  -> prompt builder                   : chat template + (codec.encode of the
                                        reference wav) spliced in on the delay
                                        staircase; produces the 33-row input grid
  -> DelayEmbeddings                  : per-row table lookup over 33 tables,
                                        SUMMED into one hidden vector per step
  -> Qwen3 backbone (+ KV cache)      : prefill the prompt, then autoregressive
                                        decode one step at a time
  -> 33 LM heads                      : text head + 32 audio heads -> 33 logit rows
  -> delay state machine              : decides, per step, which rows are active /
                                        padded (the staircase from delay_state.py)
  -> sampling                         : temp / top-k / top-p / rep-penalty,
                                        seedable (mt19937_64); or greedy
  -> (loop until audio_end / max_new_tokens)
  -> collected 32 audio-code streams  -> codec.decode -> 24 kHz waveform
  -> loudness_normalize                : RMS -> target -20 dBFS, gain clamped to
                                        [-3, +3] dB (fidelity match to upstream)

Voice cloning runs the reference wav through codec.encode and prepends those codes (on the staircase) so the backbone continues the speaker.

New src/ files and their roles

qwen3.{hpp,cpp}            one Qwen3 decoder layer: per-head q/k RMSNorm, no
                           biases, GQA, RoPE, SwiGLU FFN, RMSNorm
delay_backbone.{hpp,cpp}   stack of Qwen3 layers + KV cache (prefill + decode)
delay_embeddings.{hpp,cpp} 33-table embedding lookup, summed to one hidden/step
lm_heads.{hpp,cpp}         text head + 32 audio heads (pad code masked out)
delay_state.{hpp,cpp}      delay staircase state machine (port of delay_state.py)
sampling.{hpp,cpp}         temp/top-k/top-p/rep-penalty sampler, seedable RNG
de_tokenizer.{hpp,cpp}     Qwen3 BPE tokenizer (pre-tokenizer regex approximated)
prompt.{hpp,cpp}           chat-template prompt builder + delay splice + cloning
delay_constants.hpp        token ids, table widths (1025), pad code (1024), etc.
moss_tts_delay.{hpp,cpp}   DelayTTS orchestrator: load + tts(); the pimpl behind
                           moss::Delay in include/moss_tts.h

moss::Delay (the public API) is a thin pimpl over DelayTTS. CLI: moss-tts-cli tts --model BACKBONE --codec CODEC --tokenizer TOK --text "..." [--reference R.wav] [--seed N] [--greedy] --out OUT.wav.

Converters

• scripts/convert_moss_tts_delay_to_gguf.py -> one GGUF holding the Qwen3 backbone (qwen3.* tensors + qwen3.{hidden,n_layers,n_heads,n_kv_heads, head_dim,intermediate,rope_base,rms_eps,text_vocab} metadata), the 33 embedding tables + 33 heads (de.embed_tokens, de.emb_ext.{0..31}, de.lm_head_text, de.lm_head_audio.{0..31}), and moss.de.* metadata (n_vq, audio_vocab, sample_rate, special token ids). Run with --strict. Emits an f32 backbone (~30 GB for 8B); quantize it with the script below.
• scripts/convert_tokenizer.py -> the BPE tokenizer GGUF for de_tokenizer.
• scripts/quantize_gguf.py -> selective, allowlist-based GGUF quantizer (auto-detects --kind delay from qwen3.*/de.* tensors; also handles the Foundation codec GGUF). See Backbone quantization below.

Backbone quantization (scripts/quantize_gguf.py)

Shrinks the 8B Delay backbone (~30 GB f32 -> ~5 GB) by re-typing ONLY the Qwen3 matmul weights, which are the only backbone tensors read through ggml_mul_mat (qwen3.cpp), and ggml_mul_mat dequantizes the Q*_K/Q8_0 row formats natively.

• QUANTIZE (to --type, default q8_0): the Qwen3 backbone matmuls only — qwen3.blk.{i}.attn_{q,k,v,o}.weight, qwen3.blk.{i}.ffn_{gate,up,down}.weight.
• KEEP F32 (the #1 footgun — do NOT quantize):
  • ALL de.* tensors (de.embed_tokens, de.emb_ext.{0..31}, de.lm_head_text, de.lm_head_audio.{0..31}). DelayEmbeddings (CPU table gather) and the LM heads (CPU dot products) read these as raw float32 (host-staged at load via read_tensor_f32, then CPU gather / dot) — they are NOT routed through ggml_mul_mat, so quantizing them corrupts the reads. They must stay f32.
  • all norm/scale vectors: qwen3.blk.{i}.{attn_norm,attn_q_norm,attn_k_norm, ffn_norm}.weight and qwen3.output_norm.weight (small; full precision).
• All KV metadata is copied verbatim.

Note: pure-python K-quant encoding (q4_k/q5_k/q6_k) needs a recent gguf-py; if unavailable the script errors cleanly — use q8_0/q4_0 here, or run llama.cpp's llama-quantize on the f32 GGUF (same allowlist applies).

Tests

CI parity fixtures (model-independent, pinned by numpy or upstream-python; run anywhere):

Env-gated real-model gates (SKIP/77 without their env vars):

test_e2e_tts runs the full stack and checks the wav is 24 kHz, > 0.3 s, audible, and unclipped. test_closed_loop synthesizes "the quick brown fox..." (seed 12345), shells out to a parakeet ASR CLI (verified contract: <cli> transcribe --model M --input W -> transcript on stdout; the test is env-gated so it SKIPs without the parakeet vars), then asserts >= 70% input-word recall.

V1 gotchas (these will bite again)

1. Qwen3 q/k per-head RMSNorm. Each attention head's q and k are RMSNorm'd individually (head_dim-wide), before RoPE. Not a single norm over the whole projection. Biases are absent throughout the backbone.
2. Do NOT normalize the summed input embedding. DelayEmbeddings sums the 33 per-row table lookups into the hidden vector and feeds it straight in — no RMSNorm/scaling on the sum (the analog of the Foundation "no input norm" bug).
3. 1025-wide audio tables / pad code 1024. Each audio table/head is 1025 wide: 1024 real codes + index 1024 = the pad/delay code. Heads mask the pad code out of sampling; embeddings have a real row for it (delayed/inactive rows embed the pad code).
4. Delay staircase ported line-for-line from delay_state.py. The per-step active/padded decision is a faithful port; when it drifts, diff against the python reference (test_delay_state pins it), don't "fix" it by intuition.
5. Determinism via --seed / our mt19937_64. Same seed -> same audio on the same build. Exact bit-match against the upstream torch RNG is explicitly out of scope (different PRNG); parity is checked at the logit level (test_backbone_parity), not the sampled-token level.
6. Tokenizer pre-tokenizer regex is an approximation for non-ASCII. The Qwen3 BPE pre-tokenizer split is approximated; ASCII English matches upstream, exotic Unicode pre-token boundaries may differ. test_de_tokenizer pins the cases we rely on.
7. Loudness normalization is applied last. DelayTTS::tts runs loudness_normalize on the assembled *wav just before returning (port of pipeline.py: RMS -> -20 dBFS target, gain clamped to [-3, +3] dB). It's a fidelity match to the upstream output level, not an intelligibility step; it scales the whole waveform by one factor (skipped if empty).

Known follow-ups (V1)

• GPU ->data path for embeddings/heads — DONE. The library is now device-safe: moss::read_tensor_f32/read_tensor_i32 (in backend.hpp) host-stage the embed/head weight tables into owned std::vectors at load (the gather/dot math is unchanged → CPU byte-identical), and local_adapters to_local/head_logits use the device-safe graph-with-inputs pattern (ggml_set_input/ggml_set_output + compute_graph_with_inputs + ggml_backend_tensor_get), so src/ is grep-clean of tensor ->data and a GPU build (-DMOSS_TTS_GGML_CUDA/METAL/VULKAN=ON) is correct-by-construction. Residuals: (a) GPU execution is unverified on this CPU-only box — a real CUDA/Metal/Vulkan smoke-run is deferred to GPU hardware; (b) host-staging duplicates the f32 embed/head tables in host RAM — split-buffer placement (no duplication) is a future follow-up; (c) the category-B test-internal-ctx ggml-op unit tests (test_rope/test_transformer_block/test_transformer_masked/ test_patchify/test_layernorm_gelu/test_moss_tts_mlp/test_quantizer/ test_partial_decode/test_window_mask) still build their own CPU ctx + compute_graph and read computed outputs via ->data (most also read loader-allocated input/reference fixtures via ->data — the same loader-backed read class fixed elsewhere); they don't exercise the library's device-safety, and because the computed-output reads need the full set_input/set_output rework regardless, each is deferred as a whole-file conversion for a GPU ctest.
• Host head dot products are multi-threaded — DONE (2026-06-05). The LM/audio heads keep the deliberate host-staged f32 design (gathered/dotted on the host, not routed through ggml_mul_mat, so never quantized), but the per-step vocab-row loops in lm_heads / rt_heads / nano_heads now run through moss::parallel_for (src/parallel.{hpp,cpp}, std::thread chunked, threshold via env MOSS_HEAD_PARALLEL_MIN, default 512; the moss-tts target now links Threads::Threads). Each logit is an independent dot with its accumulation order untouched → byte-identical; the dominant text head (~311M scalar MACs/step for V1) now uses all cores instead of one. (The V2-local head_logits already runs through ggml mul_mat, so it was already threaded; the embedding gather is a cheap row memcpy, left serial.) Validated by test_parallel_for (parallel== serial + exact [0,n) coverage across edge cases) + the head parity tests at the same maxerr, including forced-parallel (MOSS_HEAD_PARALLEL_MIN=1) reruns on the tiny fixtures (test_{lm,rt,nano}_heads_parallel).
• Audio rep-penalty at penalty != 1 is untested. The repetition-penalty path exists in sampling.cpp but has only been exercised at penalty == 1.
• Pre-tokenizer non-ASCII heuristic (see gotcha 6) — revisit if a non-English prompt mis-tokenizes.

Closing V1 on real hardware

# 1. Convert the 8B Delay backbone, the codec, and the tokenizer
.venv/bin/python scripts/convert_moss_tts_delay_to_gguf.py \
    --model OpenMOSS-Team/MOSS-TTS \
    --out models/moss-tts-delay-f32.gguf --strict
#    (codec = the Foundation MOSS-Audio-Tokenizer GGUF; tokenizer via convert_tokenizer.py)
.venv/bin/python scripts/convert_tokenizer.py --model OpenMOSS-Team/MOSS-TTS \
    --out models/moss-de-tokenizer.gguf

# 2. Produce the logit-parity reference dump
.venv/bin/python scripts/gen_delay_reference.py --model OpenMOSS-Team/MOSS-TTS \
    --out /tmp/ref_dump.gguf

# 3. Wire env vars + run the env-gated gates
export MOSS_TTS_DELAY=models/moss-tts-delay-f32.gguf
export MOSS_DE_REF_DUMP=/tmp/ref_dump.gguf
export MOSS_TTS_TOKENIZER=models/moss-audio-tokenizer-f32.gguf
export MOSS_DE_TOKENIZER=models/moss-de-tokenizer.gguf
ctest --test-dir build -R 'test_backbone_parity|test_e2e_tts' --output-on-failure

# 4. Closed-loop WER gate needs a parakeet.cpp build + ASR model
export MOSS_PARAKEET_CLI=/path/to/parakeet-cli
export MOSS_PARAKEET_MODEL=/path/to/parakeet-asr.gguf
ctest --test-dir build -R test_closed_loop --output-on-failure

# 5. Benchmark TTS tokens/sec + RTF
./bench_tts.sh cpu  --model models/moss-tts-delay-f32.gguf \
    --codec models/moss-audio-tokenizer-f32.gguf --tokenizer models/moss-de-tokenizer.gguf

V2: MossTTSLocal (RQ-Transformer text-to-speech)

The second generative variant. Same text(+reference) -> speech contract as V1, but the 33 channels (1 text + 32 audio) are produced by an RQ-Transformer instead of the delay staircase: a global Qwen3 backbone runs over time, and at every timestep a small local (depth) transformer runs over the 33 channels to emit all of them for that frame. There is NO delay pattern — every frame is fully populated. Generation stops on audio_end.

Pipeline (one synthesis)

text + instruction/language + (reference wav)
  -> de.tokenizer (Qwen3 BPE)        : text -> token ids
  -> prompt_local builder             : chat template + (codec.encode of the
                                        reference wav) spliced in WITHOUT any
                                        delay; produces the 33-row input grid
  -> LocalEmbeddings (global sum)     : per-channel emb_list lookup over 33
                                        tables, SUMMED into one hidden/step
  -> global Qwen3 backbone (+KV cache): RoPE on; prefill the prompt then
                                        autoregressive decode, one timestep/step,
                                        producing a global hidden vector h_t
  -> per timestep, the DEPTH LOOP (the time x depth keystone):
       seed the local seq with h_t, then for channel c = 0..32:
         LocalAdapters.to_local       : project running local hidden -> local dim
         local (depth) transformer    : 4 layers, NO RoPE, q/k-norm KEPT; per-frame
                                        KV cache (reset() once, then step() one
                                        channel at a time -> O(channels))
         LocalAdapters.head_logits    : per-channel out-MLP + head_norm + lm_head
         sample channel c (or greedy) : audio rep-penalty 1.1 default
         LocalEmbeddings (re-embed)   : embed the just-sampled code via the SAME
                                        emb_list table and append to the local seq
  -> 32 audio-code streams            -> codec.decode -> 24 kHz waveform
  -> loudness_normalize                : RMS -> -20 dBFS (same as V1)

Voice cloning is identical in spirit to V1 (codec.encode the reference, prepend its codes), but on the flat grid — no staircase.

New / modified src/ files

qwen3.{hpp,cpp}            REUSED for BOTH transformers, gated by a `use_rope`
                           flag: global = RoPE on; local = RoPE OFF (q/k-norm
                           still applied). Same layer code, two configs.
local_transformer.{hpp,cpp}  the 4-layer no-RoPE depth transformer; per-frame KV
                           cache (reset()/step(), mirroring rt_local) -> O(channels)
local_embeddings.{hpp,cpp} the 33-table emb_list: global SUM (prompt/prefill)
                           AND single-code re-embed (inside the depth loop)
local_adapters.{hpp,cpp}   in-MLP (h_t -> local seed) + per-channel out-MLP /
                           head_norm / lm_head (to_local + head_logits)
moss_tts_mlp.hpp           the SwiGLU MossTTSMLP used by the adapters (two
                           DISTINCT-dim variants; no bias, no pre-norm)
prompt_local.{hpp,cpp}     chat-template prompt builder, NO delay splice
moss_tts_local.{hpp,cpp}   LocalTTS orchestrator: load + tts(); pimpl behind
                           moss::Local in include/moss_tts.h

The qwen3_load_layer tensor-prefix loader was generalized so the same loader reads both qwen3.blk.{i}.* (global) and local.blk.{i}.* (depth) layers. moss::Local (public API) is a thin pimpl over LocalTTS. CLI: moss-tts-cli tts-local --model LOCAL --codec CODEC --tokenizer TOK --text "..." [--reference R.wav] [--seed N] [--greedy] --out OUT.wav.

Note the local transformer has head_dim (128) != local_hidden (1536) and n_heads*head_dim != hidden — these are derived from the real q/k/v_proj tensor shapes, not assumed equal (see the converter note below).

Converter

scripts/convert_moss_tts_local_to_gguf.py -> one GGUF with three families:

• qwen3.* — the global backbone (tensors + qwen3.{hidden,n_layers, n_heads,n_kv_heads,head_dim,intermediate,rope_base,rms_eps,text_vocab}).
• local.* — the depth transformer (local.blk.{i}.*, local.output_norm). Its head_dim / n_heads / n_kv_heads are DERIVED FROM THE REAL local q/k_proj shapes (q_proj rows / head_dim = n_heads; k_proj rows / head_dim = n_kv_heads; head_dim = q_norm length). For this checkpoint: head_dim=128, n_heads=16, n_kv_heads=8, with local_hidden=1536 (so head_dim != local_hidden).
• lc.* — embeddings/adapters/heads: lc.embed.{0..32}.weight, lc.in_mlp.{gate,up,down}.weight, lc.out_mlp.{0..32}.{gate,up,down}.weight, lc.head_norm.{0..32}.weight, lc.lm_head.{0..32}.weight.

Quant allowlist (scripts/quantize_gguf.py, auto-detects the local kind from qwen3.* + local.* tensors): QUANTIZE only the matmul weights of both transformers — qwen3.blk.{i}.attn_{q,k,v,o} / ffn_{gate,up,down} and local.blk.{i}.attn_{q,k,v,o} / ffn_{gate,up,down}. KEEP F32: all lc.* tensors (read as raw float32 — host-staged via read_tensor_f32, gathered/dotted host-side — so quantizing corrupts them) and all *_norm.weight (q/k-norm, attn/ffn norm, both output_norms).

Tests

CI parity fixtures (model-independent; run anywhere):

Env-gated real-model gates (SKIP/77 without their env vars):

test_e2e_local runs the full V2 stack and checks the wav is 24 kHz, > 0.3 s, audible, and unclipped. test_closed_loop_local synthesizes "the quick brown fox..." (seed 12345), shells out to a parakeet ASR CLI (verified contract: <cli> transcribe --model M --input W -> transcript on stdout; the test is env-gated so it SKIPs without the parakeet vars), then asserts >= 70% input-word recall.

V2 gotchas (these will bite again)

1. No-RoPE local attention, but q/k-norm KEPT. The depth transformer turns RoPE OFF (use_rope=false) yet still applies per-head q/k RMSNorm. Don't drop the norms when you drop RoPE.
2. Two distinct-dim SwiGLU MossTTSMLP adapters, NO bias / NO pre-norm. The in-MLP and out-MLP have different in/out dims and carry neither a bias nor a pre-norm. The local transformer's head_dim*n_heads != local_hidden, so dims genuinely differ — do not "simplify" them to a single square MLP.
3. The emb_list is SHARED for global-sum and local-reembed. The same 33-table lc.embed.* does the prefill global SUM and the in-loop single-code re-embed. One table set, two call sites.
4. The depth loop uses a per-frame local KV cache. Each timestep reset()s the local transformer, then step()s one channel at a time (each new token attends to the cached K/V for channels 0..c), so the loop is O(channels). Don't forget the per-timestep reset() before the channel loop.
5. Audio rep-penalty 1.1 by default, seeded from per-channel prompt history. Unlike V1 (penalty 1.0), the Local audio sampler defaults to 1.1 and seeds the penalty history from each channel's prompt codes.
6. The maybe_cont seq_kv=1 fix. The continuation/decode path must use seq_kv=1 for the single new timestep; a wrong seq_kv silently corrupts the global KV-cache attention. Pinned by test_depth_loop.
7. n_heads*head_dim != hidden in the local. Don't assume the projection output width equals the hidden width — it doesn't here.

KNOWN FOLLOW-UPS (V2)

1. prompt_local.gguf fixture provenance. The test_prompt_local fixture was dumped from a faithful NUMPY RE-DERIVATION of the Local processor (torch was unavailable in the venv), not the genuine MossTTSDelayProcessor. Recommend a one-time torch-box cross-check against the real processor on the first real-model run, then re-bless the fixture if it drifts.
2. Depth-loop performance — DONE. The local depth transformer now uses a per-frame KV cache (LocalTransformer::step/reset, mirroring rt_local), so the depth loop is O(channels) instead of O(channels^2). And LocalTransformer::step / LocalAdapters::to_local / head_logits now reuse persistent scratch contexts (make_ctx_buf) instead of allocating a fresh ggml ctx per call. The change is numerically identical (validated by test_depth_loop exact codes + test_local_transformer per-row step parity). Per-step ctx-reuse is now COMPLETE across ALL per-step builders (2026-06-05): the four that still make_ctx(256 MB) every call — DelayBackbone::run, NanoBackbone::run, RtLocal::step, NanoLocal::step — now hold a std::vector<uint8_t> scratch_ sized once at load to ggml_tensor_overhead()*2*B + ggml_graph_overhead_custom(B,false) + 1 MB (B = the ggml_new_graph_custom node budget: 4096, or 8192 for NanoBackbone) and build every step's graph in it via make_ctx_buf. Byte-identical (the per-builder keystones test_delay_kv / test_nano_backbone / test_rt_depth_loop / test_nano_frame_loop pass at the same numbers); removes the per-step mmap/munmap + page-fault churn (most impactful in the depth loops, called 16–32× per frame). Steady RSS +~3–7 MB per converted instance — the tight bound deliberately avoids the 64/256 MB alternatives so Nano stays light. (The codec's full encode/decode make_graph_ctx is one-shot per clip, not per-step, so it is intentionally left as-is; the codec STREAMING step already uses make_ctx_buf.)
3. GPU ->data path — DONE. Embeddings / heads host-stage their weight tables via moss::read_tensor_f32/read_tensor_i32 at load and local_adapters uses the device-safe graph-with-inputs pattern (CPU byte-identical; src/ grep-clean of tensor ->data); a GPU build is correct-by-construction. See the V1 follow-up for the full description + residuals (GPU-unverified here; host-RAM duplication; category-B ggml-op unit tests).

Closing V2 on real hardware

# 1. Convert the Local backbone + the BPE tokenizer (codec = the Foundation's,
#    shared with V1; no separate codec conversion needed).
.venv/bin/python scripts/convert_moss_tts_local_to_gguf.py \
    --model OpenMOSS-Team/MOSS-TTS-Local-Transformer \
    --out models/moss-tts-local-f32.gguf --strict
.venv/bin/python scripts/convert_tokenizer.py --model OpenMOSS-Team/MOSS-TTS \
    --out models/moss-de-tokenizer.gguf

# 2. Produce the global+local logit-parity reference dump
.venv/bin/python scripts/gen_local_reference.py \
    --model OpenMOSS-Team/MOSS-TTS-Local-Transformer --out /tmp/local_ref.gguf

# 3. Wire env vars + run the env-gated gates
export MOSS_TTS_LOCAL=models/moss-tts-local-f32.gguf
export MOSS_LOCAL_REF_DUMP=/tmp/local_ref.gguf
export MOSS_TTS_TOKENIZER=models/moss-audio-tokenizer-f32.gguf
export MOSS_DE_TOKENIZER=models/moss-de-tokenizer.gguf
ctest --test-dir build -R 'test_local_parity|test_e2e_local' --output-on-failure

# 4. Closed-loop WER gate needs a parakeet.cpp build + ASR model
export MOSS_PARAKEET_CLI=/path/to/parakeet-cli
export MOSS_PARAKEET_MODEL=/path/to/parakeet-asr.gguf
ctest --test-dir build -R test_closed_loop_local --output-on-failure

# 5. Benchmark wall-time + RTF
./bench_local.sh cpu --model models/moss-tts-local-f32.gguf \
    --codec models/moss-audio-tokenizer-f32.gguf --tokenizer models/moss-de-tokenizer.gguf

v1.5 support (MOSS-TTS-Local-Transformer-v1.5)

Delay vs Local (the naming that ends the confusion). The V2 section above ("MossTTSLocal") describes the engine's local.arch=qwen3 path. That path is really the Delay arm: OpenMOSS-Team/MOSS-TTS-Local-Transformer (model_type: moss_tts_delay), n_vq=32, 24 kHz mono, local depth transformer = Qwen3 4-layer (no-RoPE) + in/out MLP adapters + head_norm + 1025-wide composite heads. v1.5 is a distinct arm with a materially different architecture; do not treat it as "the same Local path with flags". See the design doc docs/superpowers/specs/2026-07-10-v1_5-real-gptj-port-design.md for the full ground-truth port (supersedes the earlier speculative v1.5 notes).

Local v1.5 = OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 (model_type: moss_tts_local), n_vq=12, 48 kHz stereo. Real architecture (438 bf16 tensors + config.json, offline-verified):

• Backbone = Qwen3 36-layer (hidden 2560) under the transformer.* prefix, and transformer.embed_tokens is used (text input embed + tied text head), not frozen/skipped. The converter maps it to lc.embed.0.weight (the text input embedding table); text_lm_head is SKIPPED (tied to it). It is NOT emitted as qwen3.token_embd.
• Local depth transformer = GPT-J, not Qwen3 (gpt2_config: n_embd 2560, n_head 32 -> head_dim 80 MHA no-GQA, n_layer 1, n_inner 9728). Per block: LayerNorm with bias (ln_1/ln_2, eps 1e-6, not RMSNorm), fused c_attn QKV, interleaved RoPE (base 1e6) on q/k, scaled-dot attention (scale 1/sqrt(80)), 2-matrix silu MLP (fc_in/fc_out, not gated SwiGLU), local_transformer.ln_f as output norm. wte = nn.Identity() (fed hidden states directly). Emitted under local.* with metadata local.arch="gptj", local.norm_type="layernorm", local.rope_base=1e6, local.ln_eps=1e-6.
• Bare, weight-tied heads (no adapters): text_lm_head tied to transformer.embed_tokens; audio_lm_heads.{0..11} tied to audio_embeddings.{0..11}; local_text_lm_head[2,2560] binary head. Applied directly to the local hidden (lm_head[c] . local_out): no in/out MLP, no head_norm. _global_hidden_to_local is the identity (both 2560). Metadata flag lc.bare_heads=1 drives this C++ head path. There is NO speech_embedding_to_local_mlp, local_to_speech_embedding_mlps, or layer_norm_before_lm_heads.
• Audio vocab 1024, not 1025 (lc.audio_vocab=1024): pad code 1024 is masked to zero in the embed-sum (audio_embeds * valid_mask), never embedded. No baked pad row. Audio channels still mask the pad code to -inf in the head logits.
• Binary channel-0 decision head (lc.local_text_head_mode=1): a direct 2-wide lc.local_text_head.weight -> argmax {continue = gen_slot, stop = audio_end}. Special-token ids come from the real config (audio_start=151669, audio_end=151670, gen_slot=151656, ...) read from GGUF metadata.
• MOSS-Audio-Tokenizer-v2, 48 kHz stereo, 32 codebooks decoded at depth 12 (partial first-k decode; codec quantizer check is >=, a 32-codebook codec serves a 12-code model). moss_tts_local emits interleaved stereo; Local::channels() drives the CLI's save_wav(..., n_channels). The v2 codec decode is now parity-gated (dequant + 12 decoder stages + stereo interleave) via test_codec_v2_parity (step 5c below); the C++ codec loader is name-tolerant (v2 self_attn.in_proj / ffn.0 / ffn.2 with a legacy Foundation in_projs.0 / linear1 / linear2 fallback).

Depth loop (v1.5). 12 local positions (0..11): the local transformer runs once on the pos-0 global hidden; the binary text decision (channel 0) and audio codebook 0 both read that pos-0 hidden; codebook c reads position c; only audio_embeddings[c] feed the local transformer (no text/slot feed). to_local(global hidden) is the identity for GPT-J.

Coexistence. The converter auto-detects the arm by config shape (gpt2_config present -> v1.5 Local -> local.arch="gptj"; else Delay -> local.arch="qwen3"). The C++ loader reads local.arch once and dispatches local-block construction + forward. Absent (older GGUF) => "qwen3" => byte-identical Delay behavior. bf16 checkpoints load via a torch framework="pt" -> f32 fallback.

Offline validation (always-run, no real model): test_local_v15_engine (op-level GPT-J block + bare-head + pad-mask gate on a tiny synthetic fixture from gen_test_fixtures.py), plus the older schema fixtures test_local_v15_binary_head, test_prompt_local_v15, test_local_v15_stereo. These bind to OUR output schema, so they validate independently of the real checkpoint.

Conversion + run (user hardware, the real end-to-end gate; needs the ~5 GB v1.5 + the v2 tokenizer + torch):

# 1. Model (auto-detects the GPT-J v1.5 arm from config shape; bf16 ok;
#    --strict must report 0 unmapped source keys).
python scripts/convert_moss_tts_local_to_gguf.py \
    --model OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
    --out models/moss-tts-local-v1_5-f32.gguf --strict
# 2. Codec: MOSS-Audio-Tokenizer-v2 via the (config-driven) Nano converter.
python scripts/convert_audio_tokenizer_nano_to_gguf.py \
    --model OpenMOSS-Team/MOSS-Audio-Tokenizer-v2 \
    --out models/moss-audio-tokenizer-v2-f32.gguf --strict
# 3. Tokenizer: v1.5 ships a standard BPE; special-token ids resolve by name.
python scripts/convert_tokenizer.py \
    --src OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
    --out models/moss-tokenizer-v1_5.gguf
# 4. Synthesize (48 kHz stereo wav) + per-stage bench.
./build/bin/moss-tts-cli tts-local --model models/moss-tts-local-v1_5-f32.gguf \
    --codec models/moss-audio-tokenizer-v2-f32.gguf \
    --tokenizer models/moss-tokenizer-v1_5.gguf --text "..." --out out.wav
./build/bin/moss-tts-cli bench local --model models/moss-tts-local-v1_5-f32.gguf \
    --codec models/moss-audio-tokenizer-v2-f32.gguf \
    --tokenizer models/moss-tokenizer-v1_5.gguf --text "..."

# 5a. VERIFY offline (always-run op-level gate; no real model needed).
ctest --test-dir build -R test_local_v15_engine --output-on-failure

# 5b. VERIFY real-model (the parity gate that proves the port on real weights).
#     The dumper loads the real modeling via trust_remote_code, written for
#     transformers 4.57.1. The .venv may hold transformers 5.x, which can break it,
#     so pin 4.57 FIRST (converters are numpy/torch-only and do not need this).
.venv/bin/pip install "transformers~=4.57"
.venv/bin/python scripts/gen_local_v15_reference.py \
    --hf-model OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --out ref_v15.gguf
MOSS_TTS_LOCAL_V15=models/moss-tts-local-v1_5-f32.gguf MOSS_LOCAL_V15_REF_DUMP=ref_v15.gguf \
    ctest --test-dir build -R test_local_v15_parity --output-on-failure
#     PASS = verified. On failure, the printed per-stage maxerr (global_hidden /
#     local_text / audio / codes) localizes which stage diverged; fix + re-run.

# 5c. VERIFY the v2 codec decode against the real model (dequant + interleaved
#     waveform). Same "transformers~=4.57" pin as 5b (the dumper loads the real
#     modeling via trust_remote_code); install it first if not already done.
.venv/bin/python scripts/gen_codec_v2_reference.py \
    --hf-model OpenMOSS-Team/MOSS-Audio-Tokenizer-v2 --out ref_codec_v2.gguf
MOSS_CODEC_V2=models/moss-audio-tokenizer-v2-f32.gguf MOSS_CODEC_V2_REF=ref_codec_v2.gguf \
    ctest --test-dir build -R test_codec_v2_parity --output-on-failure
#     PASS = the RLFQ dequant + 12 decoder stages + stereo interleave match; a
#     diverging stage shows up in the per-stage maxerr.

# 6. Publish (Batch B; quantize f16/q8_0/q4_k_m + hf upload; --dry-run previews).
scripts/publish_moss_gguf.sh \
    --model models/moss-tts-local-v1_5-f32.gguf \
    --codec models/moss-audio-tokenizer-v2-f32.gguf \
    --tokenizer models/moss-tokenizer-v1_5.gguf \
    --card scripts/moss_tts_local_v1_5_card.md \
    --repo mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF

Codec-v2 decoded-audio parity (RLFQ vs our ResidualLFQ) and HF publish stay Batch B (separate go-ahead); high-confidence same family, but real-weights parity is user-gated.

V3: MossTTSRealtime (RQ-Transformer text-to-speech, offline)

The third generative variant. Same text(+reference) -> speech contract as V1/V2, and the same RQ-Transformer shape as V2 (a global Qwen3 backbone over time, a small local depth transformer over the codebooks per frame), but tuned for the realtime model: 17 channels (1 text + 16 audio), a RoPE-on depth transformer with a per-frame KV cache, per-codebook heads only, and a hierarchical text-lead prompt. The audio is decoded through the Foundation codec's first-16 partial-depth decode. This port is the offline loop — the full model run end to end; native streaming / multi-turn is a follow-on (see below). Generation stops on audio_end (EOS) on codebook 0.

Pipeline (one synthesis)

text + instruction/language + (reference wav)
  -> de.tokenizer (Qwen3 BPE)        : text -> token ids
  -> prompt_rt builder (hierarchical) : chat template + (codec.encode of the
                                        reference wav, first 16 codes/frame)
                                        spliced in; text leads the audio stream
                                        (BOS_AUDIO=1025 seeded on channel 1 at the
                                        last text position); 17-row input grid +
                                        a remaining-text stream fed one id/frame
  -> RtEmbeddings (17-sum)            : per-channel table lookup over 17 tables
                                        (rt.embed.0 text + rt.embed.1..16 audio),
                                        SUMMED into one hidden/step
  -> global Qwen3 backbone (+KV cache): RoPE on; prefill the prompt then
                                        autoregressive decode, one timestep/step,
                                        producing a global hidden h_t
  -> per timestep, the DEPTH LOOP (the time x depth keystone, rt_local):
       depth 0 input = h_t INJECTED DIRECTLY (NO projection; requires
                       global_hidden == local_hidden == 2048)
       local.reset()                  : per-frame KV cache reset each frame
       for codebook i = 0..15:
         rt_local.step(in, i)         : 4-layer RoPE depth transformer, one new
                                        position appended to the per-frame KV
                                        cache (RoPE position = i), q/k-norm kept
         rt_heads.logits(i, h)        : the per-codebook head i (no head norm, no
                                        pad mask — RT samples BOS/EOS freely)
         sample codebook i            : windowed audio rep-penalty 1.1, window 50
         RtEmbeddings.embed_local_one : re-embed the just-sampled code via LOCAL
                                        table i (the off-by-one: table i feeds
                                        depth i+1) as the next depth's input
  -> stop when codebook 0 emits EOS_AUDIO (1026); drop that final frame
  -> 16 audio-code streams            -> codec.decode(n_quantizers=16) -> 24 kHz
  -> loudness_normalize                : RMS -> -20 dBFS (same as V1/V2)

Voice cloning runs the reference wav through codec.encode, keeps the first 16 of 32 codes per frame, and splices them into the prompt grid (same spirit as V1/V2, on the hierarchical grid).

New src/ files and their roles

qwen3.{hpp,cpp}            REUSED again: global = RoPE on; the local depth
                           transformer is ALSO RoPE-on here (unlike V2's no-RoPE
                           local) — same layer code, gated by config.
rt_local.{hpp,cpp}         the 4-layer RoPE depth transformer with a PER-FRAME KV
                           cache: reset()/step(in, depth) appends one position
                           (RoPE position = depth) instead of recomputing the
                           whole channel seq.
rt_embeddings.{hpp,cpp}    the 17 global sum-embeddings (embed_sum, prefill +
                           per-frame) AND the 15 local single-code re-embeds
                           (embed_local_one, inside the depth loop).
rt_heads.{hpp,cpp}         the 16 per-codebook LM heads (bias-free, NO head norm,
                           NO pad mask — RT may sample BOS/EOS).
prompt_rt.{hpp,cpp}        the hierarchical text-lead prompt builder + cloning
                           splice; returns prefill_ids (S x 17) + remaining_text.
moss_tts_rt.{hpp,cpp}      RealtimeTTS orchestrator: load + tts(); the pimpl
                           behind moss::Realtime in include/moss_tts.h.
rt_constants.hpp           RVQ=16, CHANNELS=17, AUDIO_VOCAB=1027, AUDIO_PAD=1024,
                           BOS_AUDIO=1025, EOS_AUDIO=1026, REF_AUDIO_PAD=151654,
                           TEXT_PAD=151655, SAMPLE_RATE=24000, REP_WINDOW=50.

The codec gained a partial-depth decode (decode(..., n_quantizers=k)) so the 16 generated codebooks decode through the 32-codebook Foundation codec (first-16, the n_quantizers extension validated by test_partial_decode). moss::Realtime (public API) is a thin pimpl over RealtimeTTS. CLI: moss-tts-cli tts-rt --model RT --codec CODEC --tokenizer TOK --text "..." [--reference R.wav] [--seed N] [--greedy] [--language L] [--instruction I] --out OUT.wav.

Note depth-0 injection requires global_hidden == local_hidden == 2048 (the hidden is fed straight into the depth transformer with no adapter); the local transformer's head_dim (128) != local_hidden (2048) — head_dim/n_heads/ n_kv_heads are DERIVED FROM THE REAL local q/k_proj shapes (head_dim=128, n_heads=16, n_kv_heads=8), not assumed.

Converter

scripts/convert_moss_tts_rt_to_gguf.py -> one GGUF with four families (verified against the real checkpoint header — 403 tensors, single shard, 0 unmapped):

• qwen3.* — the global backbone (language_model.* -> tensors + qwen3.{hidden,n_layers,n_heads,n_kv_heads,head_dim,intermediate,rope_base, rms_eps,text_vocab}). The frozen language_model.embed_tokens.weight is intentionally skipped (the global input path uses the summed rt.embed.*, whose rt.embed.0 is the text path — there is no qwen3.token_embd).
• rt.embed.{0..16}.weight — the 17 global sum-embeddings (embed_tokens.{0..16}; [0]=151936x2048 text, [1..16]=1027x2048 audio).
• rtl.* — the local depth transformer (local_transformer.model.* -> rtl.blk.{i}.*, rtl.output_norm). Its head_dim/n_heads/n_kv_heads are DERIVED FROM THE REAL local q/k_proj shapes (q_proj rows / head_dim = n_heads; k_proj rows / head_dim = n_kv_heads; head_dim = q_norm length): for this checkpoint head_dim=128, n_heads=16, n_kv_heads=8, local_hidden=2048.
• rtl.embed.{0..14}.weight (15 local input embeds, 1027x2048) and rtl.head.{0..15}.weight (16 bias-free local LM heads, 1027x2048).

Plus rt.* scalars / special-token ids (rt.rvq, rt.audio_vocab, rt.audio_pad, rt.bos_audio, rt.eos_audio, rt.ref_audio_pad, rt.text_pad, rt.delay_tokens, rt.sample_rate, rt.{pad,im_start,im_end}_token_id). Run with --strict (0 unmapped against the real checkpoint).

Quant allowlist (scripts/quantize_gguf.py, auto-detects the kind from qwen3.*

• rtl.*/rt.* tensors): QUANTIZE only the matmul weights of both transformers — qwen3.blk.{i}.attn_{q,k,v,o} / ffn_{gate,up,down} and rtl.blk.{i}.attn_{q,k,v,o} / ffn_{gate,up,down} (the only tensors read through ggml_mul_mat, which dequantizes natively). KEEP F32: all rt.* / rtl.embed.* / rtl.head.* tensors (read as raw float32 — host-staged via read_tensor_f32, gathered/dotted host-side — so quantizing corrupts them) and all *_norm.weight (q/k-norm, attn/ffn norm, both output_norms).

Tests

CI parity fixtures (model-independent; run anywhere):

Env-gated real-model gates (SKIP/77 without their env vars):

| Test | Needs | | --------------------

Truncated - read the full file at https://github.com/mudler/moss-tts.cpp/blob/ee722b8e9205ee9b1b1c398a4e87e4e393e9be41/AGENTS.md.