voice

Speak to the console. The voice worker turns microphone audio into text as you talk, transcribes recordings to timestamped segments, and reads replies aloud. Speech-to-text can use bundled English models or a host-installed whisper.cpp command and multilingual GGML model; nothing leaves the machine unless you select a remote backend.

Dictation is two passes. A small streaming model produces words as you speak and decides where utterances end. The final pass can use the bundled Parakeet TDT 0.6B v2 model, a local whisper-cli process, or a remote speech backend. The selected final pass re-decodes each finished utterance with punctuation and casing; until it answers, the streaming text remains visible.

Three surfaces, one worker:

1. A mic in every chat. A Dictate button joins the composer toolbar beside attach (the chat header on consoles without that slot). Click to toggle, or hold to talk. Words appear as you speak in a header pill and land in the composer when you stop, ready to edit before sending.
2. Read aloud. Every finished turn gets a Read aloud action above the composer that speaks the last reply.
3. A voice page. The voice page shows the engine state, downloads the model, transcribes a WAV file with per-segment timestamps, and runs a dictation test.

Agents get the same through voice::* functions and the voice::transcript trigger.

Install

iii trigger compose::add worker=voice

The first dictation or transcription downloads the streaming model (about 44 MB) into data/voice/models under the project directory and loads it in well under a second; the second-pass model (about 660 MB, CC-BY-4.0) follows in the background. voice::models::download id=parakeet-tdt-0.6b-v2 fetches it ahead of time.

Direct whisper.cpp

On Voice → Overview, select whisper.cpp on this machine, choose a Whisper model, then click Download when it is missing. Tiny, base, small, medium, and large-v3-turbo are multilingual choices; the UI shows download sizes, installation state, progress, and errors. The Models section also lets you select a model with Use, even before downloading it. Model selection does not start a download. Settings offers the same picker and keeps a Custom GGML file path option for existing files.

Downloads are stored under models_dir// and checked against pinned SHA-256 digests. Save changes to the models directory before downloading from Settings; download actions use the saved directory, not the form draft. The live streaming model can also be selected and downloaded on Overview, regardless of the final transcription backend.

Set stt.backend to whisper_cpp to invoke an existing whisper-cli binary without running an HTTP server. The worker writes a temporary 16 kHz mono WAV, asks the CLI for JSON, parses its timestamped segments, and removes the temporary files. The command is executed directly, never through a shell.

stt:
  backend: whisper_cpp
  model: zipformer-en-20m       # live dictation preview and boundaries
  num_threads: 4
  whisper_cpp:
    command: whisper-cli
    model: whisper-large-v3-turbo # or /models/ggml-large-v3-turbo.bin
    language: pt
    timeout_secs: 300

Install whisper-cli separately; downloading weights does not install the command. Download a catalog model ahead of use with:

iii trigger voice::models::download id=whisper-large-v3-turbo

Whisper weights are downloaded only when requested, not during transcription. Use a multilingual GGML model for Portuguese. .en models only recognize English. voice::doctor reports a missing command or model before the first transcription. The configured language is used unless a voice::transcribe request supplies its own language value.

Quickstart

Transcribe a recording:

iii trigger voice::transcribe path=./meeting.wav

{
  "text": "The quick brown fox jumps over the lazy dog.",
  "segments": [
    { "segment": 0, "text": "The quick brown fox jumps over the lazy dog.", "start_secs": 0.3, "end_secs": 2.9 }
  ],
  "duration_secs": 5.5,
  "model": "zipformer-en-20m",
  "backend": "local"
}

Generate speech on the server and play it in the requesting browser:

• Voice → Read aloud → Local synthesis → this browser uses espeak-ng (Linux) or say (macOS) in WAV file-output mode. No sound plays on the server.
• voice::speak returns audio_base64, mime, a unique speech_id, and played: false for every enabled backend. The browser plays that response.
• Each request uses a private temporary file, removed after synthesis. Multiple clients can generate speech independently; there is no audio broadcast.
• Stop pauses and releases only this browser's audio. While generation is pending it discards the eventual response; it does not cancel other clients' requests or stop them globally. Generation may continue until completion or the local 60-second synthesis timeout.
• espeak-ng is needed only on the worker machine, not on each client. A server audio device is not needed. For Brazilian Portuguese set tts.voice: pt-br.
• Audio is returned after synthesis completes, not streamed. Local output is capped at 64 MiB in addition to tts.max_speak_chars.

For an API caller, generate audio (this command alone does not play it):

iii trigger voice::speak text="Build finished, three tests failed."

Check what the worker can do right now:

iii trigger voice::doctor

Natural local voice and voice chat

Read words, not Markdown markers

Read aloud and Voice chat convert Markdown to prose on the worker, before any TTS backend runs. **bold**, *italic*, headings, list bullets, checkboxes and table separators are not pronounced. Links keep their visible label, not the hidden URL; image descriptions and inline-code text are kept. Fenced/indented code blocks, raw HTML blocks and footnote bodies are omitted. No HTML is rendered and no link is fetched during conversion.

For example, Use **Piper** para *ler* becomes Use Piper para ler. Literal punctuation and symbols are preserved rather than deleting all asterisks or underscores: 2 * 3, snake_case and escaped markers remain literal.

The voice::speak request accepts text_format: "markdown" (default) or text_format: "plain" to preserve text literally. The chat's selected-passage button uses plain because it receives already-rendered text; full replies, automatic reading and the Voice text area use markdown. Conversion affects only the speech input, never the saved message. tts.max_speak_chars is checked against the converted text; if no readable prose remains, the request reports an empty-text error instead of generating audio for formatting symbols.

Listening and reading model configuration

The Models page separates model selection and downloads into two independent sections rather than a single Speech models list:

• Listening models · Speech to text: Zipformer (live dictation), Parakeet and Whisper (final transcription). These process microphone or recorded audio into text; translation into another language is not configured on this page.
• Reading models · Text to speech: Piper voices used by Read aloud and Voice chat to speak text in the requesting browser. The language filter is scoped to this section only; it never hides listening models.

Each section has its own installed count. Selection buttons explicitly say Use for live dictation, Use for transcription, or Use for read aloud. Choosing a listening model preserves the configured reading voice and vice versa. Download, progress, error and removal controls remain available in both groups. This is a model-configuration layout change, not a new microphone/playback mode.

Choose your language before downloading

Piper is not limited to Brazilian Portuguese. In Overview → Neural voice, Settings → Read aloud, or Models, the What language do you speak? field filters the available voices. Type a language name (português, English, español) or a regional code (pt-BR, pt-PT, en-US). Suggestions show the language/region and number of voices. Language names are searchable in English, Portuguese, the browser's locale and the language's native name.

Choose a matching voice and explicitly click Download. Names, quality, download size and installed status help distinguish the choices. A language without matches shows an empty-state explanation; it never falls back silently to Portuguese. Filtering does not change the configured voice, download files, or delete installed voices. On Models, installed Piper voices remain visible for management until a language filter is entered.

The bundled snapshot contains 176 voices across 57 language/region variants from rhasspy/piper-voices at commit 1162a9173d0ce503555aed757976b7a9912eae4c. Only metadata is embedded, not model weights. Catalog browsing works offline; a requested download needs network access. Each additional voice's ONNX, configuration and model card use pinned URLs and SHA-256 digests. The existing Faber id and installed directory remain compatible. Availability in this catalog is not a claim that every voice has been synthesis-tested; licenses and quality vary by voice, so consult its card.

Maintainers can refresh a reviewed snapshot with python3 voice/scripts/import_piper_catalog.py --revision <40-character-commit> --output . The importer reads upstream metadata, verifies small files against Git blob ids and derives their SHA-256, and takes weight hashes from LFS metadata. It does not download weights and refuses to overwrite an existing destination. Catalog updates require a worker build; browsing does not fetch a mutable remote index at runtime.

GPU preferred, CPU fallback

tts.piper.device defaults to auto: the worker requests Piper's --cuda mode (NVIDIA CUDA). ONNX Runtime may transparently use CPU when that provider is unavailable. If the CUDA-preferred process fails or times out, the worker retries once without --cuda, with the same model and text, on CPU. Auto uses up to 30 seconds per attempt, within the existing 60-second synthesis budget. If both attempts fail, both errors are reported. Files are private to each attempt and the CPU attempt never consumes partial GPU output.

Choose Settings → Read aloud → Processing device → CPU only to skip GPU and allow a single CPU attempt up to 60 seconds. This option changes Piper TTS only, not Whisper, listening models or the browser playback destination.

tts:
  backend: piper
  piper:
    device: auto # prefer GPU; use cpu to force CPU

GPU acceleration requires a compatible NVIDIA GPU/driver, CUDA/cuDNN libraries and the GPU build of ONNX Runtime (onnxruntime-gpu) inside the same Python environment/container as Piper. The regular piper-tts install uses CPU ONNX Runtime. Do not treat a GPU physically present on the host as sufficient, or install CPU and GPU ONNX Runtime packages side by side (they share the same Python module). Match package/library versions using ONNX Runtime's CUDA requirements. The worker installs no GPU runtime or drivers automatically. This CUDA path does not enable Apple Metal or AMD acceleration.

Doctor/Overview show the requested policy, not the actual provider used. The CLI does not report its effective provider in the speech response; successful auto synthesis alone is not proof of GPU usage. Model quality and voice do not change between CPU and GPU, and GPU is not guaranteed to be faster for short requests because Piper currently loads the model per request.

Install and use Piper

For a more natural voice than eSpeak, select Piper · natural local voice in Voice → Overview or Settings. Install piper-tts==1.8.0 separately on the worker (for example with uv tool install piper-tts==1.8.0 --python 3.12). The executable must be on the worker's PATH, or set tts.piper.command to its absolute path. Piper is invoked as a separate process, not linked into the worker.

For example, filter by pt-BR, download Piper Faber · Brazilian Portuguese in Models (~63 MB), then choose Use model. Other languages use the same flow. ONNX weights, configuration and the model card are SHA-256 verified. This changes TTS only: your Whisper/dictation configuration is preserved. No model downloads happen implicitly during speech generation, and missing Piper or weights produce an actionable error rather than silently using eSpeak.

tts:
  backend: piper
  piper:
    command: piper
    model: piper-pt-br-faber-medium

The chat footer has three independent controls:

• Read aloud reads the last final assistant reply.
• Read selection appears when you highlight a passage in this conversation's messages. It reads only that passage, not selections in other panes or forms. The Voice → Read aloud text area also supports selecting only a passage.
• Voice chat explicitly enables incremental reading in this browser/conversation. As assistant text arrives, complete sentences enter a local audio queue; long sentences use phrases of at most 240 characters (or the configured speech cap, whichever is smaller). It does not wait for the turn to finish. Visible progress text is included; thinking blocks, tool arguments and tool results are not read. It does not open the microphone or send a draft automatically. Enabling it does not fetch or replay chat history; enabled mid-turn, the next live snapshot can include text already generated in that current message. Duplicate/out-of-order message revisions are ignored. Completion flushes only the unread tail, not the full response again. A new turn or cancellation clears the old queue. Stop also turns automatic reading off. Switching chats, closing the view, or a playback error disables it; other clients are unaffected.

Streaming speech still synthesizes on the worker, one short text request at a time, with at most one next clip prepared ahead of the current audio. It is not sample-level TTS streaming: first-audio latency depends on receiving a complete sentence/phrase and the selected backend's synthesis speed. Snapshot preparation is coalesced over 120 ms; no history polling is used. The same CommonMark parser prepares the full growing message, preserving code-fence/link context; incomplete inline syntax (and ambiguous literal markers) may be held until more text or the final flush. A message rewrite that changes already-queued speech stops automatic reading with an error rather than repeating/replacing words already heard.

voice::speech::prepare is stateless and returns plain text plus the configured chunk cap without generating audio. The browser then queues voice::speak with text_format: plain. Queue overflow (128 waiting clips), oversized messages (256 KiB), or more than 128 message entries in one turn stops with an actionable error. Stopping discards local pending work and late audio; it cannot cancel an already-dispatched server synthesis request. No model/config changes or automatic downloads are needed.

Stop/Cancel finishes even if the microphone permission prompt remains unanswered. The browser cannot abort that prompt; if permission is granted later, the obsolete capture is immediately released without reviving dictation. Cancel during Stop clears and suppresses the result; if the remote Stop request was already sent, it cannot be retroactively changed, but its returned text is not delivered.

Chat reading controls validate the current configuration on each attempt rather than staying disabled from an old availability snapshot. After installing a voice or fixing its executable, retry Read aloud without reloading the conversation.

The browser may block automatic playback according to its autoplay policy; errors are shown rather than claiming playback succeeded. Piper loads its model per request and returns the completed WAV, so there is a generation delay. The configured tts.max_speak_chars still applies; select a shorter passage if needed.

Dictation over the bus

A dictation session is a live recognizer stream. The caller names a function that receives transcript events, then pushes 16 kHz mono 16-bit PCM in ~100 ms chunks:

iii trigger voice::dictation::start output_function_id=my::transcript
iii trigger voice::dictation::push session_id=d_… seq=1 pcm16_base64=…
iii trigger voice::dictation::stop session_id=d_…

Events carry kind (partial replaces the in-progress text, final commits a segment, closed ends the session), seq, segment, and text. The same events fan out on the voice::transcript trigger, filterable by session_id. Sessions idle for session_idle_secs are closed by the worker.

Configuration

Stored in the configuration worker under voice and editable in the console's Settings; every field takes effect on the next call, and a model or endpointing change reloads the recognizer on next use.

Field	Default	Meaning
models_dir	data/voice/models	Where models live (relative to the project directory).
stt.backend	local	local (bundled recognizer), whisper_cpp (host whisper-cli), openai (any /v1/audio/transcriptions server), or router (any speech provider registered with llm-router).
stt.model	zipformer-en-20m	Streaming model for live words and sentence boundaries: zipformer-en-20m (44 MB, fastest) or zipformer-en-large (73 MB, fewer misheard words in the preview).
stt.final_model	parakeet-tdt-0.6b-v2	Second-pass model that re-decodes each utterance for the final text. Empty disables the second pass.
stt.num_threads	2	Decoder threads.
stt.silence_after_speech_secs	0.8	Trailing silence that commits an utterance.
stt.silence_without_speech_secs	2.4	Trailing silence that ends an empty segment.
stt.max_utterance_secs	20	Longest utterance before a forced commit.
stt.openai.base_url, api_key, model, language	OpenAI defaults	The remote transcription endpoint. api_key accepts ${OPENAI_API_KEY}.
stt.router.model, language	empty	Router model (provider::model) for voice::transcribe and for the second pass of every dictated sentence; empty lets the router pick. Keys live in the router's Settings, not here.
stt.whisper_cpp.command	whisper-cli	Command on PATH, or an explicit executable path.
stt.whisper_cpp.model	whisper-large-v3-turbo	Catalog id (whisper-tiny, whisper-base, whisper-small, whisper-medium, whisper-large-v3-turbo) or a custom GGML file path. Download catalog models from the Voice page or voice::models::download; custom paths remain user-managed.
stt.whisper_cpp.language, timeout_secs	auto, 300	ISO 639-1 language hint (pt) or detection, and the per-process timeout.
tts.backend	host	piper (natural local neural voice), host (local WAV synthesis with say on macOS or espeak-ng on Linux, returned to the browser), openai (/v1/audio/speech, audio returned to the caller), router (llm-router's router::speak, any registered speech provider, audio returned to the caller), or off.
tts.piper.device	auto	Prefer NVIDIA CUDA, with CPU fallback; cpu forces CPU. Requires GPU dependencies in the Piper environment to accelerate.
tts.piper.command, tts.piper.model	piper, piper-pt-br-faber-medium	Local neural TTS executable and downloaded voice catalog id. Audio plays in the browser.
tts.voice, tts.rate_wpm	system default	Host voice and speaking rate.
tts.max_speak_chars	4000	Longest text one voice::speak call reads.
tts.openai.base_url, api_key, model, voice	OpenAI defaults	The remote speech endpoint.
tts.router.model, voice, format	empty, empty, mp3	Router model, voice id or name, and container for voice::speak.
max_audio_bytes	10485760	Largest inline (audio_base64) file for voice::transcribe.
max_sessions	8	Open dictation sessions across all callers.
session_idle_secs	120	Idle time before a session is closed.

A local whisper server (whisper.cpp server, speaches, and similar) works as the openai backend with an empty api_key.

Functions

Function	What it does
voice::transcribe	A WAV file (path or base64) to text with segments.
voice::dictation::start / push / stop / list	Live sessions.
voice::speak	Generate audio for the requesting client to play.
voice::speech::prepare	Prepare a growing Markdown snapshot without audio; returns plain text and the chunk character cap.
voice::speak::stop	Legacy compatibility: returns stopped: 0; browser playback is stopped locally.
voice::models::list / voice::models::download / voice::models::remove	Local model catalog, install, delete.
voice::doctor	Backends, model state, open sessions.

Triggers: voice::transcript, voice::session-started, voice::session-stopped, voice::model-progress. The legacy voice::speech-ended trigger remains registered for compatibility but is no longer emitted: each browser uses its audio element's ended/error events.

Limits

• The streaming Zipformer and Parakeet transcription models are English-only. Multilingual transcription uses whisper.cpp or a remote backend; Piper reading voices cover the languages listed in the catalog.
• WAV is the only container decoded. Convert other formats first, for example ffmpeg -i in.m4a -ac 1 -ar 16000 out.wav.
• Every read-aloud backend returns audio to the requesting browser; nothing plays on the worker machine. Local synthesis returns a complete WAV, not a stream.
• Prebuilt binaries cover macOS (Intel and Apple silicon) and x86_64 glibc Linux. The speech engine publishes no static no-TTS archive for Linux on aarch64, musl or 32-bit ARM, and the build only accepts archives whose SHA-256 it pins.

Licenses

The worker is Apache-2.0. THIRD_PARTY_NOTICES.md lists bundled components and the documented child-process and service integrations with their licenses and sources. If you configure a custom child-process command or model, review that artifact and its license separately. In short: sherpa-onnx and its Kaldi-derived libraries are Apache-2.0, ONNX Runtime is MIT, KISS FFT is BSD-3-Clause; the streaming Zipformer model is Apache-2.0 and Parakeet TDT 0.6B v2 is NVIDIA's under CC BY 4.0, with the attribution text in the notices file. The build links the no-TTS archives and drops espeak-ng and its ucd tables (GPL-3.0), so no GPL code is in the binary; on Linux, read-aloud runs espeak-ng as a separate program when it is installed. Models are downloaded on the worker machine from Hugging Face (ONNX transcription models may download on first use; Whisper and Piper require an explicit download), and voice::models::list reports each model's license, author and source.

Development

cd voice
cargo build                 # builds ui/ with pnpm first
cargo test
III_VOICE_UI_WATCH=1 cargo run   # hot-reloads the console UI from ui/dist

Run the recognizer against real audio with the model on disk:

VOICE_TEST_DOWNLOAD=1 VOICE_TEST_WAV=./clip.wav cargo test --test engine_live -- --nocapture