Self-Hosting the Voice Agent LLM: From Naive Config to Production Latency

Self-hosting the language model is the hard layer of the voice stack, and the one where a naive setup misleads. A first pass on stock settings comes out 2 to 3 times slower than the production bar: speech-to-text and text-to-speech clear their budgets on the same self-hosted stack, while an untuned language model’s 1.5 to 2.5 second time-to-first-token pushes the full voice-to-voice turn to 2 to 4 seconds against a target near 800 ms. The gap is serving configuration, not the model, and tuned properly the same model class lands back in the production range.

This is the second part of a three-part series on self-hosting the voice stack, between streaming speech-to-text, the clean win, and text-to-speech. It walks both ends: what a naive config produces, the fixes that close the gap, what a well-served deployment achieves, and the GPU memory math that decides which model even fits.

Where the budget actually breaks

A voice turn has a budget: reply audio should begin within roughly 800 ms of the caller stopping to feel like conversation, and past roughly 1.5 seconds callers talk over the agent or hang up. Our voice agent latency budget guide breaks the budget down component by component.

On our stack, two of the three model layers fit their slice: speech-to-text returned a final transcript in around 150 ms, text-to-speech started audio in around 200 ms. The language model’s time-to-first-token alone consumed more than the entire turn budget.

What we deployed

The agent ran a cascaded pipeline, speech-to-text into a language model into text-to-speech, orchestrated through Pipecat over a LiveKit transport. The language model was Nemotron 3 Nano 30B-A3B in BF16, served by vLLM on a single A100-80GB through Modal.

The model choice follows the logic of real-time voice. A mixture-of-experts model activates a small fraction of its weights per token: a 30B model with roughly 3B active parameters generates at the speed of a 3B dense model while holding the knowledge of a much larger one. That profile is what a latency-bound voice loop needs, and the choice of LLM for a voice agent hangs on it more than on raw benchmark rank.

The architecture splits reasoning into two roles: a fast conversational model in the voice loop answering the caller, and a slower strategist running in parallel, reading the same transcript and injecting short guidance. Only the conversational model gates the turn, so only it has to be fast.

One honest limit frames everything below: we deployed one model, not two. The alternative, Qwen3-30B-A3B, was scripted for deployment but never run, to conserve cloud credits. Every Nemotron-versus-Qwen claim in this article comes from published benchmarks, not our own head-to-head.

The naive baseline: what stock config produces

Served warm on the A100 with no tuning, Nemotron 3 Nano produced:

Metric	Measured	Target
Time-to-first-token (warm)	1,500–2,500 ms median, 25–30 s outliers	300–500 ms
Sustained generation	~14 tokens/sec	–
Peak generation (short replies)	~80 tokens/sec	–
Cold start	60–90 s	–
Voice-to-voice (warm)	2,000–4,000 ms	~800 ms

A representative five-turn session:

Turn	LLM first token	LLM total	Response tokens	TTS first audio
1 (opening)	2.56 s	2.92 s + tool retries	56	0.23 s
2	1.54 s	3.45 s	280	0.17 s
3	timeout	25.59 s	malformed	–
4	3.79 s	4.37 s	70	0.24 s
5	1.43 s	7.92 s	700	0.24 s

Most turns started generating in 1.4 to 3.8 seconds. One stalled for 25.6 seconds with no isolated root cause, the tail-latency event that turns a single bad call into a lost customer. Tool calls added their own failure: malformed JSON on the first attempt, recovered on a retry that cost roughly another second. Text-to-speech started audio in around 200 ms on every turn. These are our deployment’s numbers, not the model’s published benchmarks, and the gap between the two is the entire point.

Four causes, four fixes, in order of payoff

The model was not the bottleneck; the serving was. Three of the four causes are configuration, and each pairs with a known fix.

Full BF16 precision. The largest, slowest tier: most weight in memory, most data moved per token. Fix: an FP8 or 4-bit checkpoint roughly doubles throughput and halves the memory footprint for a sliver of accuracy. The single largest lever, and the one this deployment skipped.

Default vLLM mixture-of-experts config. The server warned on every start that it had no tuned configuration for this model on this GPU and fell back to a generic one. Community reports put the fallback cost at 3 to 5 times throughput. Fix: regenerate the tuned config, about an hour of one-time work and a few dollars of compute. We never did it.

Reasoning mode on by default. The model emits a hidden chain-of-thought block before its answer, pure latency the caller waits through. Before the flag was set, the model read its internal monologue aloud through text-to-speech. Fix: a no-think parameter on every request; there is no separate non-reasoning checkpoint, only a runtime toggle.

Single-stream serving with cold starts. One warm stream never exercises the batching that makes self-hosting economical, and a 60 to 90 second cold start makes the first caller after a scale-down wait through a timeout. Fix: continuous batching plus a warm pool.

One more fix has no matching cause above: right-size the GPU and the quant together. A 4-bit model with an FP8 KV cache fits a single L4. Avoid splitting a model across two L4s; the card has no fast interconnect, and PCIe transfer adds 5 to 15 ms per token, which undoes the latency goal.

The model was fine. The serving config was not.

• Quantize: move off BF16 to FP8 or 4-bit for speed and memory.
• Tune the mixture-of-experts config: recover the 3-5x throughput vLLM left behind.
• Disable reasoning per request: stop paying for tokens the caller never hears.
• Batch and keep warm: serve many streams per GPU, and hold the model resident.

None of this is research. The distance between a capable model and a production voice agent is measured in engineering days, not model quality.

Stuck getting a self-hosted model under the voice latency bar? Softcery does the quantization, serving, and latency engineering that decides whether self-hosting is viable. Schedule a consultation to audit your stack.

What proper serving achieves

The fixes change the numbers by an order of magnitude. Quantizing off BF16 roughly doubles throughput, the mixture-of-experts config tune recovers 3 to 5 times more on top, and batching turns one stream into many. Published throughput for a quantized 30B-A3B model runs 68 to 94 tokens per second (Artificial Analysis, DeepInfra), against the 14 tokens per second the naive stock config produced.

Time-to-first-token is the metric voice lives on. Published voice benchmarks show where a well-served small model actually lands:

Model class, well-served	Voice-loop TTFT	Source
8B dense (Llama-3.1-8B)	~300 ms	Cerebrium
30B-A3B MoE (Nemotron 3 Super)	687 ms	Daily.co voice eval
30B-A3B MoE (Nemotron 3 Nano)	940 ms	Daily.co voice eval

Proper serving takes the naive 1.5 to 2.5 second first token down to the 300 to 950 ms range, and model size is the lever. An 8B-class model clears the 300 to 500 ms voice target; a 30B-A3B mixture-of-experts sits at the high end, workable but tight. A hosted API like Gemini Flash still often wins on pure first-token latency. Self-hosting earns its place on cost at scale, fine-tuning, and data control, not on beating a managed API’s first token.

The memory math that decides the GPU

The most expensive mistake in self-hosting one of these models is assuming it fits a GPU it does not. Mixture-of-experts models are deceptive: active parameters drive speed, total parameters drive memory, because every expert must be resident in VRAM. A “3B active” model still needs the full 30B in memory.

Model and format	VRAM	Fits a 24 GB L4?
Qwen3-30B-A3B FP8	~31 GB	No
Qwen3-30B-A3B 4-bit	~17 GB	Yes
Nemotron 3 Nano NVFP4	~19 GB	Yes, but see below
Nemotron 3 Nano FP8	~33 GB	No
Nemotron 3 Nano BF16	~60 GB+	No, needs an 80 GB card

Two traps hide in the table. An FP8 checkpoint of a 30B mixture-of-experts model runs 31 to 33 GB and overflows a 24 GB L4, so FP8 is not the path to the cheapest capable card. NVFP4 fits at around 19 GB, but its tensor-core acceleration is Blackwell-only; on an Ada-generation L4 it loads through a software fallback that keeps the memory saving and gives back none of the speed.

One configuration detail that costs hours when wrong: the vLLM tool-call parser. Qwen3-30B-A3B Instruct uses the hermes parser. Nemotron 3 Nano uses qwen3_coder plus a custom reasoning-parser plugin. The intuitive guesses, a “qwen25” or model-named parser, do not exist or do not apply.

Picking the model: benchmarks lie about voice

Published benchmarks make Nemotron 3 Nano the clear agentic winner, but the widely cited head-to-head compares it against Qwen3-30B-A3B in its Thinking variant, not the Instruct variant a latency-bound voice agent would ship. Nemotron wins the Berkeley Function-Calling Leaderboard v4 (53.8 against 46.4), Arena-Hard v2, and the τ²-bench average narrowly, plus roughly 3.3 times the throughput in NVIDIA’s own lab. On the two τ²-bench scenarios closest to a customer-service voice agent, retail and airline, Qwen3 scores higher in NVIDIA’s own table.

The benchmark that matters for voice reorders the field. On Daily.co’s voice-agent evaluation, which scores turn-taking, tool use, and instruction-following over a 30-turn call:

Model	Pass rate	Median time-to-first-token
Nemotron 3 Super (120B-A12B)	97.0%	687 ms
GPT-4.1 (reference)	96.3%	–
Qwen3.5-27B	94.3%	–
Nemotron 3 Nano 30B-A3B	90.6%	940 ms

The model we ran posts the highest time-to-first-token among the leaders at 940 ms, the number that hurts most in a conversation, and the Instruct model from the published head-to-head was not tested at all. Leaderboard rank and voice performance are different questions. The design target is time-to-first-token under load with reliable tool-calling on the agent’s own domain: a production voice turn targets 500 to 800 ms end-to-end, the language model gets 300 to 500 ms of it, and a 940 ms first token spends the whole turn before text-to-speech starts.

The open-model landscape has since moved toward exactly the profile voice needs, small mixture-of-experts and hybrid Mamba-Transformer models that generate fast and fit one card:

Single-L4 (24 GB) candidates, mid-2026

• GLM-4.7-Flash (30B-A3B): reported to run on one 24 GB GPU at 60-80 tokens/sec, built for coding and agents.
• Gemma 4 26B-A4B: fits at 4-bit, native function calling, with multi-token-prediction variants that cut decode latency.
• Qwen3-30B-A3B-Instruct: the battle-tested baseline, stable in production since mid-2025.
• Granite 4.0 H-Tiny (7B/1B): a hybrid Mamba model whose memory barely grows with context, an excellent ultra-low-latency router.
• Nemotron Nano 9B v2 and Ministral 8B: small dense models that fit comfortably and call tools well.

An A100-80GB opens up GLM-4.5-Air, GPT-OSS-120B, and Nemotron 3 Super, which posted the 97% voice-eval result above. Across all of them, the rule for voice is firm: avoid reasoning and thinking variants. Chain-of-thought traces wreck first-token latency, and the caller hears silence while the model thinks.

The two-role split stays the cheaper architecture: keep the fast conversational model self-hosted, run the slower strategist on a cloud API. Gemini 3 Flash costs around $0.50 per million input tokens with a 90% caching discount, roughly a cent per voice session for the strategist role. Self-hosting that second model would burn a whole extra GPU container to save almost nothing.

The case this evaluation did not reach: fine-tuning

The strongest argument for self-hosting the language model is the one this deployment stopped short of. A hosted API serves one general model to everyone; a self-hosted model can be trained on a specific agent’s transcripts and tool schemas. Even a small adapter trained on a few hundred correctly-labeled tool calls tends to fix the malformed-JSON failures a stock model produces.

The tradeoff: mixture-of-experts models are harder to fine-tune than dense ones. The routing layers can collapse under a naive low-rank adapter, so the practical path is a full supervised pass or a more careful adapter method. Fine-tuning also flips the cost argument. A smaller fine-tuned model matches a larger general one on a narrow task and is cheaper to serve at every concurrency level. The general model that loses on latency and cost in this article becomes a specialized model that wins on both, after training work a hosted API removes the option to do.

When self-hosting the LLM is worth it

The language model is the hardest layer of the voice stack to self-host. Bring it in-house last.

The per-session economics make the threshold concrete. At low concurrency, a self-hosted model on an A100 costs more per session than a cloud API, because the GPU bills by the hour whether it serves three streams or three hundred. The crossover sits near several dozen concurrent sessions. Below it, a managed API like Gemini Flash wins on both latency and cost; above it, with the serving work done, self-hosting pulls ahead by a widening margin.

Self-host the conversational LLM when

• Concurrent traffic is high enough to keep a GPU saturated and batched.
• The team can invest in quantization, mixture-of-experts tuning, and serving optimization, not just vllm serve.
• A fine-tuned or domain-specific model is on the roadmap, which a closed API cannot deliver.
• Data control or vendor independence is a hard requirement.

For most teams the sequence is speech-to-text first, text-to-speech second, the language model last, kept on an API until traffic and engineering capacity both justify the move. The model we measured was capable. The serving work, not the model download, is the real project.

Softcery builds production voice agents and the serving infrastructure under them, including the latency engineering that decides whether a self-hosted model is viable. For a specific architecture and the cost model behind it, schedule a consultation.

Frequently Asked Questions

Can you self-host an open LLM for a real-time voice agent?

Yes, but it is the hardest layer to get right. A naive stock config produces a 1.5 to 2.5 second time-to-first-token, 2 to 3 times over the production bar. Tuned, the same model class lands in the 300 to 950 ms range depending on size (per published voice benchmarks): an 8B model clears the 300 to 500 ms voice target, a 30B-A3B MoE sits at the high end. Getting there takes quantization, mixture-of-experts serving tuning, and batching. On stock settings at full BF16 precision, an untuned deployment produced a 1.5 to 2.5 second time-to-first-token, which pushed the full voice-to-voice turn to 2 to 4 seconds against a production bar near 800 ms.

Why was the self-hosted LLM so slow?

Four causes, three of them configuration rather than the model: full BF16 precision instead of a quantized format, a default vLLM mixture-of-experts config that left an estimated 3 to 5 times throughput unused, reasoning mode on by default emitting hidden tokens before any answer, and single-stream serving with 60 to 90 second cold starts. Each is fixable in the serving layer.

Does a 30B model fit on a single 24 GB L4 GPU?

Only at 4-bit. A Qwen3-30B-A3B FP8 checkpoint is about 31 GB and does not fit a 24 GB L4. A 4-bit quant is roughly 17 to 19 GB and fits with an FP8 KV cache. NVFP4 also fits in memory, but its tensor-core acceleration is Blackwell-only, so on an Ada-generation L4 it loads as a software fallback with the memory saving and no speed gain.

Is self-hosting the LLM cheaper than a cloud API?

Not at low volume. Per session, a self-hosted model on an A100 can cost more than a cloud API like Gemini Flash until the GPU is heavily utilized. The crossover sits near several dozen concurrent sessions. Below that, an API wins on both latency and cost; above it, self-hosting wins by a widening margin.

Which open-source LLM should a voice agent use in 2026?

For a single 24 GB GPU, the strong picks are 30B-class mixture-of-experts models at 4-bit (GLM-4.7-Flash, Gemma 4 26B-A4B, Qwen3-30B-A3B-Instruct) or hybrid Mamba-Transformer small models (Granite 4.0 H-Tiny, Nemotron Nano 9B v2, Ministral 8B). Avoid reasoning and thinking variants for voice, since chain-of-thought tokens wreck first-token latency.

Can an AI voice agent run fully on-premise?

Yes. On-premise AI voice agents run all three layers on owned GPUs: speech-to-text and text-to-speech self-host cleanly, and the LLM works after quantization and serving tuning, with a 24 GB L4 as the floor for a 30B model at 4-bit. Data control is the usual driver, since no audio or transcript leaves the infrastructure.