dream-tts

Local text-to-speech in Rust. Four speech models ported from PyTorch — three that clone a voice from a ten-second clip, one that does not — validated stage by stage against fp32 activations dumped from their references. A 1612-word chapter becomes eleven minutes of speech in about 1m 15s with a cloned voice, or twelve minutes in 27 seconds without. One process, no Python at runtime.

Source on GitHub · Apache-2.0 · macOS on Apple silicon

The same chapter, narrated by each engine

1612 words of prose — examples/chapter.txt in the repo — rendered on one M4 / 16 GB laptop, each engine in the configuration it ships in. These are the complete chapters — eleven to thirteen minutes each — in two voices, cloned from two reference clips of about ten seconds. The first three are ordered by how they sound on this passage, best first — a listening judgement, not a measured one, so trust your own ears over it. kokoro comes last because it is not in that comparison at all: it cannot clone, so its two voices are its own.

A chapter becomes 11 minutes of speech in about 1m 15s in a cloned voice, on a laptop — 9× faster than realtime, so a 16-hour book is about 1.7 hours of compute rather than 11.5. Without cloning it is 27 seconds.

Try it

curl -fsSL https://raw.githubusercontent.com/drmhse/dream-tts/main/install.sh | sh
cd dream-tts && ./scripts/bootstrap.sh

./dream-tts speak --text-file examples/chapter.txt --out chapter.wav

curl is the only prerequisite — no Rust toolchain and no Python. The default engine, qwen3tts, has no conversion step, and a tagged release publishes prebuilt Apple-silicon binaries. bootstrap.sh downloads the checkpoint (~4.3 GB, resumable, sha256-verified) and the correctness fixtures. Add the others with bootstrap.sh audio8 cosyvoice kokoro — each converts its checkpoint with PyTorch, which is why none of them is the default. kokoro is the cheap one at ~0.7 GB. --list prints the ids and what each costs.

./scripts/bootstrap.sh kokoro
./dream-tts speak --engine kokoro --set voice=am_michael \
    --text-file examples/chapter.txt --out chapter.wav

kokoro takes --set voice=<name> rather than --voice, because there is no asset to point at: the 28 voices live in the checkpoint. dream-tts engines lists the ones installed.

Narrate a whole book

dream-tts import book.epub --out prep/ --dry-run  # see the split first
dream-tts book book.epub --out narration          # kick off; Ctrl-C is safe
dream-tts jobs                                    # what is running, anywhere
dream-tts jobs <id> --control pause --now         # stop within seconds

A thousand-page reference book is hours of synthesis, and that length changes what the software has to be. A run has to survive interruption, so the state does not live in the process doing the work: dream-tts book submits to dream-tts-serve, which owns the engine and the queue. A run has to be discoverable by a process that did not start it, so job records are files — dream-tts jobs answers with no server up, which is exactly the moment someone is about to start a second run on top of the first. Resume is the same command: a job's id is a hash of the text and the settings, so re-running adopts the chapters that finished.

Chapters come from the document itself — the EPUB spine, the DOCX outline levels, the heading levels in HTML or Markdown, the outline a PDF declares. Markdown is rewritten before it is spoken, because a *** that reaches the voice is read aloud as “asterisk, asterisk, asterisk”; about forty rules cover code, LaTeX, units, currency, dates and tables, each one arrived at by listening to a failure.

Every command, once

The whole CLI is eight verbs. Anything that reads a document (speak --text-file included) takes EPUB, DOCX, ODT, HTML, Markdown, PDF or plain text; a .txt file is spoken literally, and --raw forces that for anything.

# speak
dream-tts speak --text "Hello from a fresh install." --out hello.wav
dream-tts speak --text-file chapter.md --out chapter.wav
dream-tts speak --text-file book.epub --engine cosyvoice \
    --voice voices/my-voice --seed 7 --out book.wav
dream-tts speak --text-file narration.txt --raw --greedy --out exact.wav

# narrate a document, and watch or steer the run
dream-tts import book.epub --out prep/ --dry-run   # the split, before hours of synthesis
dream-tts import book.epub --out prep/
dream-tts book book.epub --out narration           # Ctrl-C is safe; it keeps running
dream-tts book book.epub --out narration --detach  # submit and exit
dream-tts book book.epub --out narration           # the same command resumes it
dream-tts jobs                                     # every run on this machine
dream-tts jobs <id>                                # attach to one and watch it
dream-tts jobs <id> --control pause                # stop after the chapter in flight
dream-tts jobs <id> --control pause --now          # stop within seconds, lose that chapter
dream-tts jobs <id> --control resume
dream-tts jobs <id> --control cancel

# markdown to the text an engine should speak, without synthesising
dream-tts narrate chapter.md --stats
dream-tts narrate prep/*.md --out-dir text/ --maps

# what is installed, what is configured, what is on disk
dream-tts engines
dream-tts voice voices/cosy-default-qwen3tts
dream-tts config
dream-tts storage
./scripts/uninstall.sh                             # reports; deletes nothing unnamed

A voice is a directory, and building one takes a ten-second clip and its exact transcript. It is the one step that wants Python, and it runs once per voice — the runtime loads the exported conditioning and never carries a speaker encoder.

references/qwen3tts/.venv/bin/python references/qwen3tts/export_voice.py \
    --model references/qwen3tts/weights --audio clip.wav \
    --text "the exact words spoken in the clip" \
    --name my-voice --out voices/my-voice

Or run it as an HTTP API

dream-tts-serve loads one engine once and answers requests from it — no per-request model load, which is where the 3.0 second start beats a Python service’s 15–17. It is wire-compatible with the FastAPI service it replaces, so an existing client switches by pointing at a different port.

DREAM_TTS_API_KEY=secret ./dream-tts-serve --port 3003

A missing key is a misconfiguration rather than a licence to run open, so the write routes answer 503 until one is set. It travels as X-API-Key or Authorization: Bearer; the read routes (/health, /v1/capabilities, GET /v1/jobs) need neither.

Synthesize POST /tts

curl -X POST localhost:3003/tts -H 'X-API-Key: secret' \
     -H 'content-type: application/json' \
     -d '{"text":"Hello from Rust.","voice":"voices/cosy-default-male","seed":7}' \
     -o out.wav -D headers.txt

# prose straight from a file, no JSON to quote and no python3 on the path
curl -X POST localhost:3003/tts -H 'X-API-Key: secret' \
     -H 'content-type: text/plain' -H 'X-Voice: voices/my-voice' -H 'X-Seed: 7' \
     --data-binary @chapter.txt -o chapter.wav

voice and seed are per request: the first picks a voice asset without restarting, the second makes a render reproducible. The text/plain form exists because quoting arbitrary prose into JSON from bash is a bug waiting for an apostrophe — X-Voice and X-Seed carry what the JSON fields would have.

Stream while it renders POST /tts/stream

curl -N -X POST localhost:3003/tts/stream -H 'X-API-Key: secret' \
     -H 'content-type: application/json' -d '{"text":"Hello from Rust."}' \
   | ffplay -autoexit -nodisp -f s16le -ar 24000 -ac 1 -i -

Raw PCM as each segment lands, so first audio arrives after one segment rather than after the whole render — 2.7s against 5.5s on the same text. Sample rate is the engine's own; /v1/capabilities names it in sampleRate (24 kHz for the default, 44.1 for audio8). It is slower overall on purpose: one segment at a time gives up the cross-segment batching worth 2.7× on book-length text, which is the right trade for a listener and the wrong one for a book.

Ask what it is GET /v1/capabilities, /health, /

curl -s localhost:3003/health
curl -s localhost:3003/v1/capabilities | jq '{engine, sampleRate, languages, modes}'
curl -s 'localhost:3003/?format=json' | jq          # live engine, voice, settings file
open http://localhost:3003/                          # the same thing as a page

/health answers only once the model is resident, so a reply means ready. GET / negotiates on Accept — JSON to a client, a self-contained page to a browser, with no CDN and no asset routes, because a local service that needs the network to explain itself is broken in exactly the situation someone is most likely to be reading it.

Narration runs /v1/jobs

# submit: narration text files in reading order, which `dream-tts book` writes to text/
curl -X POST localhost:3003/v1/jobs -H 'X-API-Key: secret' \
     -H 'content-type: application/json' -d '{
       "source": "book.epub",
       "texts": ["narration/text/chapter-001.txt", "narration/text/chapter-002.txt"],
       "out_dir": "narration",
       "voice": "voices/my-voice",
       "seed": 7
     }'

curl -s localhost:3003/v1/jobs | jq '{active, jobs: [.jobs[] | {id, state, source}]}'
curl -s localhost:3003/v1/jobs/<id> | jq '.chapters[] | {name, state, seconds}'
curl -N localhost:3003/v1/jobs/<id>/events         # per-segment progress, as SSE

curl -X POST localhost:3003/v1/jobs/<id>/pause     -H 'X-API-Key: secret'
curl -X POST localhost:3003/v1/jobs/<id>/pause-now -H 'X-API-Key: secret'
curl -X POST localhost:3003/v1/jobs/<id>/resume    -H 'X-API-Key: secret'
curl -X POST localhost:3003/v1/jobs/<id>/cancel    -H 'X-API-Key: secret'

Submitting the same document again is the resume: the id is a hash of the text and the settings, so the second submission finds the first run and adopts its finished chapters. The stream opens with the current state, so a watcher that attaches late is not blank until the next segment finishes. Every control answers with what it will actually do — pause keeps the chapter in flight, pause-now discards it and says so, and pausing something that cannot be paused is a 409 rather than a silent no-op.

The routes, in one table

Two things worth knowing. Every response carries its own cost — x-audio-seconds, x-wall-seconds, x-rtf, and x-stages with the per-stage split, so a client can see where the time went without a second request. And voice and seed are per request: the first picks a voice asset without restarting, the second makes a render reproducible.

What it refuses is as deliberate as what it serves. speed other than 1.0, mode=instruct and mode=cross_lingual answer 501 with an explanation rather than quietly returning something else — returning speed-1.0 audio to a client that asked for 1.5 would report the request as honoured when it was not. So do the Python service’s /v1/tts-jobs and /v1/alignment-jobs, which this one replaces with /v1/jobs rather than emulates. Synthesis is serialised behind a semaphore, because two requests interleaving on one Metal queue make both slower and neither faster.

How they compare

Compare wall time, not just RTF: the four do not produce the same duration from the same text — kokoro’s male voice speaks slowest, 13:32 against audio8’s 11:34 — and RTF divides by audio produced, which flatters a slower-speaking engine. qwen3tts leads the cloning engines by this much only because it batches across sections, so it wants length: on a 132-word passage it is 0.314, against 0.544 for audio8 and 0.716 for cosyvoice. Never benchmark it on a paragraph. kokoro has nothing to batch and is steady at any length — 0.036 on that same passage.

Memory is the other axis, and it runs the other way. On that short passage kokoro peaks at 1.3 GB, cosyvoice at 5.0 and audio8 at 9.7, against qwen3tts’s 6.7 — and 9.2 on a chapter. 16 GB is what the default engine wants. A smaller machine is better served by a different engine than by a smaller weight format.

What makes it fast

Every green component is measured. Together they took the engine to RTF 0.144–0.148 — 6.8× to 6.9× realtime; a second round since, not drawn here, reached 0.104–0.114 on the chapter, about 9×.

Four of them carry most of it, and two are Metal kernels written here. f16 weights, because only a dense GEMM shares one weight read across lanes. The codec’s convolutions gather their taps inside the GEMM — both conv-as-GEMM routes were losing to assembling the operand rather than multiplying it, 85.8 ms to build the im2col against 8.2 for the GEMM consuming it — which is 1.70× on a codec chunk, and bit-identical output. A GEMM shaped like a decode step, since candle reaches 3.64 TFLOP/s on a 2048-cube but 1.07–2.34 at m = 48. And 48 batched lanes, sorted longest-first, shedding each finished tail: a group runs as long as its longest lane, so ordering decides how much of the batch is real work — 68–70% of lane-steps were useful before, 79–90% now.

Past 48 lanes it swaps, and that is a floor rather than a saturation: a lane costs 13 MB of peak footprint, while the codec’s activations cost gigabytes. The memory analysis is in the README — it is candle’s buffer pool, not the lane count.

What it is validated against

Every engine is gated against per-stage fp32 activations dumped from its PyTorch reference, so a failure localises to a stage rather than to “the audio sounds wrong”. audio8’s greedy generation is bit-identical to its reference; cosyvoice’s teacher-forced argmax matches 105/105. kokoro has no sampler at all, so every deterministic stage is compared directly — and its one stochastic stage, the decoder’s excitation, is judged by a criterion the reference sets itself: the phase accumulates to 165,303 radians, where one f32 ulp is 0.0156, so upstream does not reproduce against itself. The port lands 25.3 dB from the reference where the reference lands 19.7 and 20.7 dB from its own other draws.

Free-running sampled output is deliberately not gated on equality — ras_sampling draws from torch’s generator, so a sampled sequence is not reproducible across implementations. Quality is checked separately by word error rate. Known limitations, including an undiagnosed regression in two convolution stages, are in the README.