How to Build a Real-Time Speech Translator

I built Parrots because I got tired of reading English subtitles in a meeting.

What this post is about. This post is about the design of Parrots, a speech translator that runs entirely on your own machine. You will see how it cuts a sentence into clauses so that you hear the translation while you are still talking, and how it makes your laptop look like a microphone to the meeting app. I will also explain where the latency budget actually went.

What you need to know first. Three terms carry most of the weight. A speech recogniser turns your voice into text. A translation model turns that text into the other language. A text-to-speech model turns the translated text back into audio. Parrots chains the three together, and everything runs on-device, so no audio leaves your computer and there is no server in the middle. The code is in Rust, a compiled language that checks its types before the program runs; you do not need to know Rust to follow this post.

The scenario is specific: I speak Chinese, the other side speaks English, and the meeting app is Zoom or Teams or 腾讯会议. The existing answer is a cloud speech-to-speech translator, and it fails for two reasons at once.

It fails on latency. A turn-based translator forces the conversation into a “one speaks, waits, reads” rhythm. My design notes recorded the number I refused to accept: three to four seconds of dead air after a long sentence. In a meeting that is a broken conversation, not a delay.

It fails on identity. The other side hears a synthetic stranger. Turn-taking collapses, because your synthetic voice carries none of the timing cues that mark a human ending a thought.

So the requirements are two constraints that pull against each other: sub-second latency, and your voice on the other end. Everything else in this post follows from those two.

Why the design looks like this

Everything runs on-device. No audio leaves the machine. That buys latency (no per-token network jitter on the critical path), privacy (confidential meeting content never goes to a third party), and offline use. It also constrains everything else: if it must be local, the models must be small enough to be fast, which is why the stack is SenseVoice-int8 (a small speech recogniser in 8-bit compressed form), opus-mt-int8 (a small translation model) and an optional Qwen-Q4 (a small language model) rather than anything frontier-sized.

Traits instead of one concrete pipeline. The design doc states the rule plainly: substitutions happen under the trait — opus-mt vs Qwen for translation, ZipVoice vs F5-TTS for synthesis (both text-to-speech models), whisper vs zipformer for recognition (both speech recognisers). The pipeline crate — a crate is Rust’s unit of packaging, one library — depends only on core‘s traits, never on a model.

It paid off: the text polisher was later added as a new trait plus an Option<Arc<dyn TextPolisher>> field without restructuring the pipeline.

Translation starts clause by clause, not sentence by sentence. A clause is the part of a sentence between two punctuation marks. This is the core latency idea: you start hearing the translation while you are still talking.

A virtual audio device. This one is forced by the platform. A meeting app takes audio from a named input device: there is no “inject PCM here” API — PCM is raw digital audio samples — no loopback API, which would be a way of feeding a program its own output, that a third-party app may use, and screen capture is not an audio path. To be heard, Parrots must appear to Zoom as a microphone. To hear the other side, it must appear as the app’s speaker. Hence two CoreAudio HAL devices — CoreAudio is macOS’s audio system, and HAL its plug-in layer for third-party devices: Parrots Microphone (2 channels) and Parrots Speakers (16).

They must be separate devices: if the translated voice and the remote voice shared one pipe, the translated voice would feed back into the captured path — a loop. The differing channel counts only keep the two names unambiguous to users and automation.

How it works

The live path is three stages with different concurrency shapes:

1
2
3
4
5
6
7
8
┌────────────────┐   512-sample chunks   ┌───────────────────┐
│ capture thread │ ────────────────────► │ segmentation task │
│ (blocking, VAD)│    chan depth 32      │ rolling ASR +     │
└────────────────┘                       │ clause commit     │
                                         └────────┬──────────┘
                              clause chan depth 2
                                                  ▼
                                         worker pool (MT → TTS → sink, single consumer) ──► virtual mic / speakers

The diagram is the live audio path. The capture thread on the left fills a 32-deep queue of 512-sample chunks; VAD is voice activity detection, deciding whether someone is speaking. The segmentation task in the middle cuts that stream into finished clauses through a queue only two clauses deep, running a rolling ASR (automatic speech recognition re-run on the growing audio window, so partial text exists early). The worker pool on the right translates (MT, machine translation) and speaks (TTS, text-to-speech) one clause at a time, then hands the audio to the virtual microphone and speakers.

The capture thread is blocking because CoreAudio callbacks are — blocking means the thread waits instead of handing the work elsewhere; the segmentation task is async, meaning it waits for a result without holding up a thread, because it waits on ASR; the worker pool has a single consumer because the TTS session is shared and cannot be used concurrently.

The segmenter

The segmenter decides where to cut the audio: a pure-logic state machine with the ASR injected as a closure — a function passed in from outside — which is why it can be unit-tested with canned text and no model files. Its thresholds are the heart of the project:

Rule Value Why
Punctuation trigger 。！？；，,!?;. followed by more text Natural clause boundary; requires more text so trailing punctuation does not trigger
Micro-pause trigger ≥400 ms RMS silence The speaker paused; treat it as a boundary
Min chars before a pause commit ≥6 new characters Stops stray interjections from being chopped
Min clause audio ≥1 s Below this, the ASR is not even called
Min content chars at a cut ≥2 (tail ≥1) Prevents punctuation-only fragments, which cause downstream MT hallucinations
Trailing punctuation never counts Rolling ASR speculatively appends a period to a partial window

Here are the same thresholds as the constants the code actually uses:

1
2
3
4
5
6
const MIN_CLAUSE_SAMPLES: usize = 16000;      // 1s of 16 kHz audio
const PAUSE_FRAME_SAMPLES: usize = 1600;      // 100ms frame
const PAUSE_MIN_FRAMES: usize = 4;            // 4 frames = 400ms
const PAUSE_SILENCE_RMS: f32 = 0.01;          // silence threshold
const MIN_CHARS_BETWEEN_COMMITS: usize = 6;   // ignore stray interjections
const MIN_CLAUSE_CONTENT_CHARS: usize = 2;    // no punctuation-only clauses

MIN_CLAUSE_SAMPLES is 16000 samples at 16 kHz — one second of audio. PAUSE_MIN_FRAMES is 4 frames of 100 ms, giving the 400 ms micro-pause trigger. PAUSE_SILENCE_RMS is the loudness below which a frame counts as silence; RMS is a plain average of signal strength, not its peak.

The decision itself:

1
2
3
4
5
6
7
8
if let Some(cut_chars) = last_committable_punct(&new_text) {
    let estimate = proportion(cut_chars, total_chars, total);
    let cut_samples = snap_to_silence(&self.window, estimate).unwrap_or(estimate);
    if cut_samples >= MIN_CLAUSE_SAMPLES
        && content_chars(&new_text.chars().take(cut_chars).collect::<String>())
            >= MIN_CLAUSE_CONTENT_CHARS
    { return Ok(Some(self.commit(&new_text, cut_chars, cut_samples))); }
}

This is the punctuation trigger. last_committable_punct finds the last punctuation mark that is not the buffer’s final character, proportion turns that position into an estimate of where in the audio the cut should fall, and snap_to_silence moves the estimate to the nearest real pause within 0.5 seconds. The estimate is crude: if the transcript is 40 characters and the punctuation sits at character 20, the cut is assumed halfway. Without the snap, a clause slice can bleed the first syllable of the next clause.

The trailing-punctuation exclusion is one line: the search covers only (0..chars.len().saturating_sub(1)), excluding the last character, because rolling ASR speculatively appends a period to a partial window.

The audio path

The capture buffer is a hand-rolled lock-free single-producer single-consumer ring — a fixed-size array where one side writes and the other reads, with no lock between them. Writing one sample is a single function:

1
2
3
4
5
6
7
8
pub(crate) fn push(&self, value: f32) {
    let h = self.head.load(Ordering::Relaxed); let t = self.tail.load(Ordering::Acquire);
    if h - t == self.cap {
        self.tail.fetch_max(h - self.cap + 1, Ordering::AcqRel);
    }
    self.slots[(h % self.cap) as usize].store(value.to_bits(), Ordering::Relaxed);
    self.head.store(h + 1, Ordering::Release);
}

head and tail are two counters into the array: the producer writes at head, the consumer reads at tail, and when they meet the buffer is full, so fetch_max moves the tail one slot forward and the oldest sample is dropped. The hand-rolled ring exists because the usual “advance head, write slot, advance tail” order transiently empties the buffer, so a reader can observe a gap; the single atomic fetch_max makes “drop the oldest sample” one indivisible step.

Echo cancellation uses the system’s own VoiceProcessingIO unit — the same AEC/AGC/NS path FaceTime uses, where AEC is acoustic echo cancellation, AGC is automatic gain control, and NS is noise suppression. The capture path tries to open that unit first, and when it fails it logs a warning and falls back to plain capture, with the playback gate taking over the echo problem. When AEC is off, the gate works like this: the translated audio physically leaves the speaker a few hundred milliseconds after it is written, so capture must be muted for written sample duration + 300 ms acoustic margin, not until the write returns.

Correcting what the ASR heard

ASR mis-hears proper nouns: 西游记 becomes 稀有記. Two layers fix it: a source bias (injecting the term list into whisper’s decoder prompt, that is, the context text the decoder is given) and post-processing that compares toneless pinyin, the phonetic spelling of Chinese without tone marks:

1
2
3
4
5
6
7
if let Some(seq) = &self.pinyin_seqs[ti] {
    let seq_same_len = &text_pinyin[start..start + tlen];
    if seq.len() == tlen
    && seq_same_len.iter().zip(seq)
        .all(|(tp, sp)| tp.as_deref() == Some(sp.as_str()))
    { candidates.push(Candidate { start, len: tlen, term }); }
}

The outer if means the term list has a pinyin sequence for this position at all. The comparison walks the recognised text and the term’s pinyin side by side, requiring the same length and the same characters (as_deref unwraps the optional string before the equality test). Tone differences do not matter, and traditional/simplified variants are covered for free, because the comparison happens after conversion to pinyin. crates/pipeline/src/hotwords.rs:113.

The trait seam

The abstraction that makes all of this testable is four small traits. A trait here is a list of methods a type promises to have, so a caller can be written against the promise instead of a specific model:

1
2
3
4
5
6
7
8
#[async_trait::async_trait]
pub trait AsrEngine: Send + Sync {
    fn supported_langs(&self) -> &[Lang];
    async fn transcribe(&self, audio: &AudioSegment) -> Result<Transcript>;
    /// Rolling-window partial transcript, not provided by default
    fn partial_stream(&self, _stream: AudioSegment)
        -> Option<broadcast::Receiver<TranscriptUpdate>> { None }
}

supported_langs reports which languages the engine handles, transcribe returns the final transcript for one piece of audio, and partial_stream returns a stream of in-progress results for the user interface — with a default implementation of None, so an engine that cannot do partials does not have to write one. crates/core/src/engine_traits.rs:4. Translator, Synthesizer and TextPolisher follow the same shape, and AudioPlatform abstracts capture, playback and AEC so that pipeline has no macOS dependency at all.

Trade-offs

The ASR backend was chosen by benchmark, not by taste. The original design picked whisper. Then an A/B gate was written — a test that lets one option win only if it beats the other on stated metrics: “SenseVoice CER must be <= whisper’s AND latency must be <= 1/2 of whisper”. CER is the character error rate, the share of characters the recogniser got wrong. SenseVoice-small measured a Chinese CER of 0.026 against whisper’s 0.385, at 3–6× the speed, running on the CPU so it does not fight whisper for Metal, Apple’s GPU interface. The cost: SenseVoice has no prompt-injection API, so it only gets the post-processing layer of hotword correction — biasing recognition toward a known list of terms.

The translation model is sentence-isolated. opus-mt is small, fast and CPU-only, which is exactly why it was chosen — it does not contend with the GPU for TTS. The price is no dialogue context: pronouns and terminology that depend on previous turns do not survive. The documented upgrade path was a small Qwen, which instead became the optional polisher.

The polisher is optional because it costs latency. --polish auto means on for files, off for live, with a hard 800 ms deadline in live mode. It fixes homophone typos (wrong characters that sound right) and removes filler words, sitting between ASR and MT, driven by a system prompt that instructs the model to correct same-sound typos against a domain glossary, delete filler words, and change nothing else.

The driver is a fork, not from scratch. The spec rejected writing my own CoreAudio HAL plugin outright: “coreaudiod 稳定性: HAL 插件崩溃会影响系统音频, 故选择经过海量验证的 BlackHole 代码基” — roughly, a HAL plugin crash takes the whole system audio down with it, so the project chose the well-tested BlackHole code base. So driver/macos is a branded BlackHole fork — 4,628 lines of C — built twice with different device names and channel counts, and the build script fails if the string “BlackHole” survives in the artifacts.

What I would do differently

I missed the latency target, and I know why. The design target was a 1.5-second first clause. Measured: 3.2–3.5 seconds. The pipeline overhead is about 1.1 seconds (segmenter tick ≤0.5 s, rolling ASR ~0.3 s, MT ~0.35 s); the rest is ZipVoice, whose real-time factor (RTF) is about 1.2 — it needs roughly 1.2 seconds of compute per second of audio.

The interesting part is why TTS is slow, because my first hypothesis was wrong. I assumed the sampling step count was the cost, so I benchmarked it: two steps gives a 451 ms RTF of 0.24, four steps gives 869 ms at 0.51. But the real cause is that the rolling voiceprint — a stored numeric description of the target speaker’s voice — changes the prompt file every clause, so the prompt cache always misses and the ONNX session (the loaded runtime instance of the model) is rebuilt every time. The fix is not a smaller step count; it is a stable prompt, or a cache key that does not change per clause.

The models are twice the size I planned. The target was ~700 MB on disk and ≤1.5 GB of RAM. Reality is about 2 GB, because whisper-small (466 MB) is retained alongside SenseVoice (155 MB) so the fallback stays available, plus the ZipVoice int8 build, its vocoder (the second model that renders synthesised speech as audio), two opus-mt pairs, and the optional 1 GB Qwen.

My documentation drifted from the code. The README says “≥6 content characters” before a pause commit, but the code checks total new characters including punctuation; the content-character rule applies only to the cut floor. The README also claims latency gates run in CI — the only workflow in the repository is a release build, there is no test CI at all. And the docs still reference a file, live_inc.rs, that was refactored away months ago.

One acceptance criterion is still open. The plan calls the remaining work “真人麦克风路径未由人亲测” — the real microphone path has not been verified by a human. The .pkg install, hot-plug reconnection (unplugging and replugging the virtual device), AEC against real speaker bleed, and the 20-second continuous-speech no-drop check are all still on the list.

The project itself is in better shape than that list suggests: there are no todo!() or unimplemented!() markers in the Rust sources, the driver builds and installs, the CLI and the menubar app both work, and the segmenter has fourteen unit tests. What remains is mostly human validation and TTS speed — but “mostly human validation” is still not validated, and I would rather say so than let the README imply otherwise.

References

  • sherpa-onnx — a sibling project from the same team, an ONNX-based speech toolkit
  • whisper.cpp — a C/C++ implementation of Whisper, the model family kept as the fallback recogniser
  • candle — a Rust machine-learning framework for running small models locally
  • ZipVoice — the text-to-speech model this post benchmarks, and the source of the voice-cloning prompt
  • BlackHole — the macOS virtual audio driver whose code base this project forks
  • CoreAudio — Apple’s audio documentation, including the plug-in layer the virtual devices sit on
  • cpal — a Rust library for cross-platform audio input and output