"Will fine-tuning help?" always arrives with a symptom attached: the agent mishears SKUs, mispronounces the brand, drifts off script by turn twelve.

Our first reply is always the same request. Send ten failing calls, each with four artifacts: the caller's audio, the transcript, the model's response text, and the synthesized audio the caller heard.

Most questions resolve there. In about 95% of production deployments a voice agent is not one model: it is a cascading pipeline of at least three, plus an orchestration layer that owns turn-taking. Each has its own fine-tuning pass, and its own set of controls that reach the same result without training anything.

Settle one thing before any of it: training a model usually means running it. A few managed platforms still tune and serve on your behalf, but that path keeps narrowing, and the open-weights route that remains brings GPUs, deployment, monitoring, and requalification every time the base model moves. Fine-tuning is a self-hosting decision as much as a modelling one. Budget for the infrastructure, not the training run.

Route the symptom to the wrong layer and you spend six figures training a model that was never at fault.

Finding the layer that produced the symptom

Lay the four artifacts side by side and you will see which layer failed.

Once you know the layer, the route follows:

Turn-taking runs on proprietary models that managed platforms do not expose for customization, so an agent that interrupts callers cannot be fine-tuned into politeness. Endpointing and barge-in settings are the only levers.

Latency belongs to no single layer. It accumulates across all of them, so measure the budget before attributing it to a model.

The cheaper answer is a faster hosted model plus the optimizations around it: stream every stage, cut the prompt, drop reasoning mode, serve from a nearer region, remove the retry that fires on every slow call. Those ship in days and reverse with a config change.

Self-hosting a smaller tuned model buys latency too, and it is the last option to reach for. You take on GPUs, serving, and requalification to win milliseconds the optimizations above often deliver on their own. Reach for it once you have exhausted them and the budget is still blown, or when volume makes owning the serving path pay for itself.

Speech-to-text: what the agent mishears

Fix it without training

Start with term boosting. Deepgram takes up to 100 terms capped at 500 tokens per request, Speechmatics up to 1,000 words or phrases per job, AssemblyAI streaming up to 100 keyterms.

Then reach for prompting. AssemblyAI's Universal-3 Pro line accepts a plain-English prompt parameter that shapes transcription as it happens, not after. It reaches what a term list cannot express: domain framing, spelling patterns generalized from two or three examples, verbatim versus clean output, number formatting, and preserving code-switched speech instead of silently translating it.

Alphanumerics are an application problem, and no recognizer solves them. Policy numbers, order codes, and email addresses fail on a clean line with a perfect model, because the caller mumbles, the digits sound alike, and a single character ruins the lookup.

Take them off the audio channel instead. Spell-back confirmation and DTMF entry handle the simple cases. Beyond that, the call does not have to carry the data at all: send an SMS link mid-conversation and let the caller type the order number, confirm the address, or upload the photo while the agent stays on the line. Voice keeps the conversation, the screen takes the precision.

You have moved the failure out of the transcript. A value typed into a form arrives validated, and nothing about it depends on how the recognizer heard it.

The fine-tuning pass

Acoustic or domain adaptation is the most expensive pass of the three.

NVIDIA recommends "on the order of 100 hours of speech or more." That is roughly 1,200 five-minute calls, transcribed word by word with correct domain spellings. You either have that archive or you do not.

Gains scale inversely with how good the base model already is on your audio. NVIDIA fine-tuned its streaming Nemotron 3.5 ASR model on Greek and Bulgarian, then measured on held-out FLEURS at the 80ms chunk size, the most demanding streaming condition with no lookahead. Greek moved from 35% to 24% word error rate, Bulgarian from 22% to 15%, both around a third better in relative terms.

Read the pattern, not the numbers. Those languages started weak, which is where the headroom was. A base model already at 6% on your traffic has far less to give, and the fine-tune that moves it a fraction of a point buys nothing a caller notices.

The language model: how the agent decides and speaks

Fix it without training

Restructure the prompt before rewriting the model. Identity and guardrails first, examples last, variable data at the end. The ordering decides whether the cache ever hits. Prompt caching keys on an exact prefix match, so anything variable near the front of the prompt costs you the cache on every turn. Read the cached-token count out of the usage payload before accepting any argument that the prompt is expensive.

Tuning a hosted base model does not reduce time to first token. Only a shorter prompt, a smaller model, fewer retries, or a different serving path does. Our voice prompt engineering guide covers what changes when output is spoken.

Automated prompt optimizers go further, searching the prompt the way training searches weights. GEPA, a Genetic-Pareto optimizer, runs the system, reflects on the traces in plain language, proposes prompt edits, and keeps whatever survives against its own best attempts. Across four tasks it beat GRPO reinforcement learning by 10% on average and up to 20%, using as much as 35 times fewer rollouts.

Try it before committing to a training run. It needs what training needs, an evaluation set and a metric, and it returns a prompt you can read, diff, and roll back.

If a clean prompt still produces inconsistency, add a second model as a copilot. It follows the conversation alongside the primary agent and corrects it mid-call: the qualifying question that never got asked, the flow step skipped, the drift off script. Corrections feed into the next turn, and its behavior changes with a prompt edit rather than a retraining cycle.

A copilot runs a turn behind, so it corrects course rather than preventing an action. Anything that must not happen at all, a tool call on missing arguments being the obvious one, needs deterministic validation in the call path instead. Reserve that blocking check for the turns that carry real risk, since it lands in the caller's silence, and measure it against your latency budget.

The fine-tuning pass

Supervised fine-tuning on curated conversations earns its cost in two situations.

Small models with weak tool judgment. In our production work, a small mixture-of-experts model struggled to decide when to ask a clarifying question rather than proceed with a tool call – expected at roughly 3.5B active parameters. Domain fine-tuning was mandatory. This applies only after schema and validation fixes have failed, and only to teams who chose a small model for latency or unit economics.

Conversational tone. Published work found that fine-tuning a small open-weights model on 100 synthetic samples outperformed system prompting for target style. One hundred samples is a realistic dataset, which makes this the cheapest training pass of the three layers.

A 10,000-token system prompt does not qualify. That is the textbook cache-hit case, and the argument for shrinking it is reliability, not cost.

When the decision lands on training this layer, the LLM fine-tune guide covers the work step by step: the dataset, the base model, and the release gate.

Text-to-speech: how the agent sounds

You fix most of this on the runtime text path, without touching a model.

Fix it without training

Change the base model first. Synthesis models improve fast, and a newer snapshot often clears a backlog the dictionary was being written to patch. Evaluate against a fixed phrase list, not by ear on three examples.

Then attach a dictionary for what remains, and keep it separate from numeric normalization, since the two fail differently. Prefer aliases and respellings over phoneme markup, which fails quietly. Check model support first: dictionary support is per-model rather than per-vendor, and models that lack it skip phoneme tags and use the default pronunciation. A dictionary that changed nothing is usually a model mismatch.

For a voice that isn't yours, the voice library and voice design come first. When they miss, clone one. Gradients move inside the vendor's pipeline, but nothing expensive lands on you: no dataset engineering, no retraining cycle. You upload samples and get a voice back, which puts cloning on this side of the decision rather than the training side.

Cloning costs you control. Vendors restrict creation to the speaker's own voice, block export of the result, and freeze it at creation, so accent and tone cannot be adjusted afterward.

The fine-tuning pass

Few teams need this. It means a managed custom neural voice, or a fine-tuned open-weight model on your own infrastructure.

The open-weight route means owning the training and serving stack, which is the self-hosting decision. Take it when cloning cannot reach the target: an accent or language variant no vendor ships, or a data boundary that forbids sending audio out at all.

Proving the tuned model is better

Our rule of thumb: no baseline, no claim. Measure the system you have before touching it, on calls the training will never see, and keep those numbers. Often that measurement ends the project on its own, because it shows the runtime control already fixed the problem.

A significant result can still be worth nothing. A fraction of a point off word error rate is real and invisible to the caller, so ask what changes on the business side before shipping it. Our testing guide covers the methodology.

Start with ten calls

Localize the layer from the four artifacts, apply that layer's runtime control, then split live traffic and read the business outcome. Consider training after those steps, or not at all.

In our experience the question resolves at the first one. Nobody gave the recognizer the vocabulary, nobody wrote the transcription prompt, the dictionary went to a model that ignores it, or nothing was storing the four artifacts.

We build and tune production voice agents, including the cases where training pays. Book a consultation and we will map which layer emits your symptom and whether tuning it returns anything at your volume.