How to Fine-Tune a Voice Agent: The STT Layer

The speech layer of a voice agent carries both the cheapest accuracy lever in the stack and its heaviest training pass. Keyterm boosting, step zero in every pipeline before any training, ships in hours and needs no dataset. The fine-tune itself needs hours of well-labelled audio, a served endpoint, and a measured regression on traffic the tune never targeted – and the audio is the hard part, because the archive worth training on comes from real calls, ideally from an agent that has already run in production for a while. Two situations justify that pass: an error rate on a specific language, accent population or domain that no configuration lever moves, and a self-hosted recogniser that has to reach the managed providers' accuracy to keep what it was deployed for, whether that was lower latency or a lower per-minute cost – and that second case only opens when the traffic and the managed invoice are large enough for owning the stack to pay.

What follows is how we run that pass in production.

Why benchmark accuracy drops on real calls

Published error rates come from clean, wideband, read speech, and a production agent hears none of that. The phone network delivers 8 kHz narrowband mono, the caller is on speaker in a car, the sentence arrives with disfluencies, restarts and an accent the test set underrepresents, and the frequency band the benchmark audio had was never captured on the line, so no upsampling restores it. Part of the gap is not the model at all: a voice-activity detector that clips the first word, an endpoint that fires early, a noise-suppression stage the telephony path did not need. That is why a failing call is worth replaying through the recogniser offline before anyone blames the weights – a transcript that comes back correct points at the pipeline, and no fine-tune fixes a pipeline. The baseline that matters is the model's error rate on your own recorded calls, and that number, not the leaderboard one, is what the rest of this article moves.

What to exhaust before training

Every serious provider ships accuracy levers that need no dataset, and they resolve most vocabulary complaints. Keyword boosting comes first: a list of domain terms the recogniser favors at decode time – products, case types, the carrier list. The list works best dynamic rather than global, because the agent knows which stage of the conversation it is in and can swap vocabularies mid-call: intake terms during qualification, SKUs during ordering, street names only while an address is being captured. Deepgram's Flux updates keyterms mid-stream without reconnecting. Boosting carries one real failure mode – pushed too hard, the recogniser starts hearing the boosted terms in audio that never contained them, so measure insertions alongside recall.

Prompting is the newer lever. AssemblyAI's Universal-3 Pro line takes a plain-English prompt that shapes transcription as it happens: domain framing, spelling patterns, number formatting, keeping code-switched speech in its original language. It reaches behavior no term list can express, and it updates turn by turn with conversation state.

Some failures no configuration reaches, and no fine-tune either. Alphanumerics – order codes, policy numbers, emails – fail with a perfect recogniser, because one misheard character voids the lookup; take them off the audio channel with read-back, DTMF or a mid-call SMS. And when the vocabulary is genuinely huge, self-hosting raises the boosting ceiling by an order of magnitude before any weights move. For a ten-thousand-term catalogue the sequence is bias, then self-host and keep biasing, then adapt. Two of those three rungs need no dataset.

Choosing a base model to adapt

Adaptation buys domain vocabulary the base model never saw, robustness on an accent population outside its training distribution, and accuracy on a noisy or band-limited channel. It buys nothing for turn-taking, which lives in orchestration, nothing factual, since no model at this layer knows anything, and no exemption from confirmation on high-risk fields.

Filter on license first, streaming second, adaptation path third; accuracy sorts whatever survives, and a public leaderboard answers none of the three. The bar a candidate has to clear is a first-party fine-tuning path with published results behind it, and in mid-2026 two families clear it. Whisper is not one of them: it processes audio in a 30-second sliding window, so the streaming in every tutorial is chunking wrapped around an offline model, and the fine-tune produces a recording transcriber rather than a live recogniser.

Nemotron 3.5 ASR Streaming 0.6B (OpenMDW-1.1, native chunks from 80 to 1120 ms) is the base we default to, for an operational reason rather than a leaderboard position: the streaming architecture we measured in production, the first-party recipe, and the serving path come from the same place and demonstrably fit. NVIDIA's tutorial starts from the released checkpoint and a manifest of audio, transcripts and language tags, keeps the pretrained tokenizer below roughly 50 hours of data, and evaluates with the same cache-aware streaming script that serves the model, at the deployed chunk size. Its worked example on Greek and Bulgarian carries the lesson we read into every data plan: a two-thousand-hour mix cut both languages' error by about a third, and a second tranche of in-language speech more than halved Bulgarian's – data moved the result further than any recipe adjustment.

Qwen3-ASR 0.6B and 1.7B (Apache-2.0) is the strongest alternative, with the caveat that streaming runs only through the vLLM backend. The official repo ships a full SFT script, and the proof is independent: Polyglot-Lion fine-tuned both sizes on roughly 970 hours of Singaporean speech using the script's default configuration, on a single GPU for about $81 of compute, and cut the 0.6B's average error across twelve local benchmarks from around 51% to 17%. The balance sits in the same literature: on a single-speaker dysarthric corpus a Qwen3-ASR fine-tune underperformed a tuned Whisper, which is why the bake-off on a sealed slice of your own traffic stays mandatory whatever the papers say.

The managed route narrowed rather than closed – most vendors now customize on text only, which reaches vocabulary and does nothing for accents, noise or codec, while audio-level training survives mainly at Azure and behind enterprise tiers elsewhere. Adaptation therefore usually means serving the recogniser, with its own cost and break-even.

The dependency also runs the other way. A team already serving its own recogniser, whether for latency or because managed per-minute pricing stops making sense at volume, starts from an open model that often trails the managed providers on accuracy, and the tune is what closes that gap. The target is concrete: the managed provider's word and character error rate, measured on a held-out set of the team's own calls, reached at the chunk size the agent serves. Until the tuned model meets that bar, whatever self-hosting saved on the invoice or the latency budget is being paid back in transcription errors; once it does, the win stands on its own.

Building the data to fine-tune STT for a voice agent

Our labeling pipeline is the same in every project: collect real calls or emulate them, draft the transcripts with an offline state-of-the-art model, confirm with human review. Offline models carry none of the streaming constraint – they hear the whole call with full context instead of an 80 ms chunk, so their drafts land closer to right than wrong – and the Whisper-class models ruled out as serving candidates earn their keep here as labelers. The human pass is review rather than transcription: listen to the audio, correct the draft, approve. The streaming model ends up training on labels a better offline model wrote.

Production already generates cheaper labels, and most stacks throw them away. A call whose order number was confirmed against the order record, or whose intake field a human corrected, carries a verified transcript at no cost; capturing it needs only the audio, the transcript and the correction event joined by one key. Instrument that join from day one whether or not a tune ever happens – an automated call evaluation pipeline produces the signal.

On volume, vendor guidance runs from thirty minutes to several hundred hours, so treat it as a learning curve rather than a threshold, and count speakers as well as minutes: ten hours from two callers is not ten hours of diversity, a constraint that shaped the Spanish-dialect coverage in CaseGen's bilingual intake. Three rules do not bend. Split held-out data by whole call and caller, or speaker identity leaks across train and test. Treat the audio itself as the sensitive record – a voice is a biometric identifier, so a redacted transcript de-identifies nothing, and regulated teams run the program under existing authorisation rather than claiming anonymity. And match the training audio to the deployed path, same codec and sample rate, with augmentation – noise, codec simulation – on the training side only.

Without an archive, three sources in order of fidelity. Synthesized speech from in-domain text, which measurably works – an Interspeech 2023 study reached a 36% relative improvement training on synthetic audio alone, and an AWS walkthrough from March 2026 adapts Parakeet for clinical dictation entirely on speech synthesized from generated medical text with hospital-noise overlays. Then role-play calls through the real carrier and codec, which produce a transcript correct by construction. Then the default we recommend: launch on a strong hosted recogniser, log everything, tune later. Synthetic audio trains; it never evaluates. The evaluation set stays real calls only.

What the tune degrades

Adaptation redistributes error: accuracy moves onto the target domain and off everything else, and the size of that move is what most write-ups omit. NVIDIA published the extreme case – a full fine-tune that lifted in-domain accuracy from 60% to 96% while general WER went from 5% to "nearly 30-50%, making the model completely unusable". The subtler version hides inside a healthy aggregate: overall WER improves while one slice the training never covered – a second language, voicemail audio, an accent population – quietly collapses. For bilingual agents the collapse is measured, with base-language WER nearly tripling after a tune on new-language data alone, and nothing in the training loop reports it, because the forgotten language is not in the training set.

The defenses are documented and cheap. Blend general speech back into the training mix – NVIDIA's Nemotron recipe calls it replay – rather than training only on the failing calls. Train several candidates and pick on target-domain gain against off-target degradation, not the first run that converges. And keep a regression set that predates the tune, covering every language, accent and channel the agent serves; without it, none of this damage is visible until production finds it.

Proving the tuned model is better

Accuracy and latency are one check, run at the configuration the agent actually serves. A tune can improve word error rate and finalize later, or behave differently at the deployed chunk size, and an offline evaluation shows neither – which is why NVIDIA measured its own streaming fine-tune at the 80 ms chunk with no future-audio peeking. The measurement rules hold unchanged: change one thing, count conversations rather than utterances, set the bar before looking.

The delta is easy to fake and easy to miss. WER counts insertions, deletions and substitutions against a reference – (I + D + S) / N over the reference word count, with CER the same arithmetic at character level – so anything that changes surface form without changing words poisons it. Both sides of every comparison go through the same normalization first:

Then check the improvement survives statistics, because the floor is higher than intuition says.

A paired comparison on the same utterances decides whether the delta is real. Braintrust's published attempt to separate major STT providers on 240 clips ended with every confidence interval overlapping every other – while a targeted vocabulary fix on the same set did clear a paired significance test. That is the shape a real fine-tune result has: a focused intervention measured paired, not a broad average drifting down. A two-point drop on twenty test calls proves nothing.

We score every test call on two levels, and neither is aggregate WER. First, entity survival. Pull the critical entities out of each reference transcript and measure the fraction that arrive intact – Braintrust's harness weights digits highest, then proper nouns and long domain words, and keeps the top few per clip, since a digit error is never cosmetic. At this level "70" against "seventy" stays an error, because the downstream parser cares about the form, while entity matching itself normalizes only lightly – lowercase, punctuation to spaces – so casing never fakes a miss. Second, answer equivalence. An LLM judge reads reference and hypothesis side by side and scores whether the difference would have changed the agent's next response – equivalent, partially changed, or different. Most transcription errors change nothing, and this metric finds the ones that do; Braintrust validated the judge against deterministic scorers before trusting it, a step worth copying. Aggregate WER sits underneath as the sanity check, not the verdict. Three turns show why the levels disagree:

The second row is the tune's whole job; the third is the noise WER spends its budget on. Latency needs the same paranoia: start the clock at the wall-clock moment the last audio byte streamed, not when the request returned, or the number silently absorbs rate-limit backoffs and upload-then-poll flows that have nothing to do with the model. The tuned model wins when entity survival and answer equivalence move on the sealed set, base and tuned judged on the same utterances, and the test harness that replays those calls is the same one that will catch the next regression.

Rollback is a routing change, not a retrain. Keep the previous model loaded beside the new one on the same inference server and route away on trigger – at roughly 2 GB of GPU memory for a 0.6B model, on the architecture we measured, that is cheap enough to be the default. The latency budget the finalization has to fit inside is a prerequisite, not a follow-up.

Where to start

The sequence is the article in one line: exhaust configuration, pick a streaming base with a proven recipe, build the dataset from real calls, and release only what beats the baseline on a sealed set. We scope STT tunes in exactly that order, including whether adaptation returns anything at your volume. Book a consultation and we will work through it with your call archive.