Self-Hosting Speech-to-Text for a Production Voice Agent: What We Measured
Last updated on July 14, 2026
We build and advise on production AI systems. Bring your questions to a free intro call.
Talk to usA self-hosted streaming speech-to-text model matched commercial accuracy, beat commercial latency, and cut the per-minute transcription cost by more than 30 times at saturation. One open model, permissive license, fast enough to test on a consumer gaming GPU. This article covers the metric that governs a voice turn, the model that fit, the architecture that scales it to hundreds of concurrent calls, and the numbers we measured against the numbers NVIDIA publishes.
This is the first part of a three-part series on self-hosting the voice stack, alongside the language model and text-to-speech. Speech-to-text comes first because it is the cleanest win: the model is small, the integration is contained, and the results are unambiguous.
The metric that governs the turn: end-of-speech to final transcript
A voice agent feels responsive or sluggish based on one gap: the silence between a caller finishing a sentence and the agent starting to reply. The number that matters is the time from end-of-speech to the final transcript, because the final transcript triggers the language model. Everything downstream waits on it.
Interim transcripts, the partial words that stream in while a person is still talking, gate nothing. They serve live captions and barge-in detection, and their timing follows the physics of articulation rather than server speed. Instrument and optimize the end-of-speech to final number; it sits inside the larger turn-gap budget covered in our voice agent latency budget guide.
Why move speech-to-text off a cloud API
A cloud STT API is the correct first choice for an early-stage voice agent. Three costs accumulate as the product scales:
- Per-minute billing grows linearly with traffic. A voice agent that succeeds turns its own success into a rising variable cost with no ceiling.
- A third-party owns the transcription layer. Provider incidents make the agent go deaf. Pricing, retention, model versions, and data handling sit outside the team’s control.
- The network round-trip is baked into every turn. Audio crossing the internet to the provider and back adds 50 to 150 ms per utterance before the model does any work. Self-hosting removes that hop.
None of these matter at ten concurrent sessions. All of them matter at five hundred.
Choosing the model: why Nemotron Speech Streaming beat Riva and Parakeet
Two wrong turns preceded the right model, and both recur in every NVIDIA speech project.
NVIDIA Riva, the official production ASR server, runs older models. New architectures need TensorRT packaging, which lands one to three months behind the open release, and the roughly 10 GB container expects an NGC account plus the NVIDIA container toolkit. Raw NeMo with Parakeet TDT, where the newest models appear first, tops the offline leaderboard but streams through a buffered path with roughly two seconds of right-context latency. Two seconds of dead air after every sentence disqualifies it for a voice agent.
The fit is NVIDIA’s 600M cache-aware streaming model built for real-time voice agents on Pipecat. The English-only 0.6B shipped 5 January 2026; the multilingual Nemotron 3.5 ASR Streaming successor, covering 40 locales on the same architecture, followed in June and is the model to start on today.
- Architecture: cache-aware FastConformer-RNNT, 600M parameters, purpose-built for streaming rather than offline transcription adapted after the fact.
- Coverage: 40 language locales, the multilingual successor to the January 2026 English-only 0.6B. Same architecture and size, so the English measurements below carry over closely.
- Input: 16 kHz, 16-bit, mono PCM. Selectable latency modes at 80, 160, 560, and 1120 ms of look-ahead, tunable at runtime.
- License: open weights, cleared for commercial use.
- Reference repo:
pipecat-ai/nemotron-january-2026ships a WebSocket server and a Pipecat client for the family, so most of the integration is already written.
The streaming-versus-offline distinction is the whole point. On the Open ASR Leaderboard (word error rate, lower is better), the streaming column separates a voice-agent model from a transcription model:
| Model | Avg WER | Streaming | Note |
|---|---|---|---|
| Nemotron Speech Streaming 0.6B | 6.93–8.43% | Native, cache-aware | Purpose-built for voice agents |
| Parakeet TDT 0.6B v2 | 6.05% | Buffered (~2 s right-context) | Top offline, unusable for real-time |
| Parakeet CTC 1.1B | 6.68–7.40% | Yes | The model classic Riva serves |
| Canary-Qwen 2.5B | 5.63% | No, batch only | Best accuracy, far slower |
| Whisper Large v3 | 7.44% | No, autoregressive | 15–50x slower than the Parakeet family |
Nemotron trades a fraction of a point of accuracy against the offline leaders for native low-latency streaming. The correct trade for a conversation.
One model, loaded once: the architecture that scales
Loading a model copy inside every worker process does not survive contact with scale: at 500 concurrent callers, 500 copies at ~1.2 GB each in half precision demand roughly 600 GB of GPU memory, each serving a single stream with no batching.
The architecture that works is a centralized inference server. The model loads once on a GPU host; voice workers stream audio in and transcripts out over a WebSocket. Capacity scales horizontally by adding GPU hosts behind a load balancer. For a model this small, NVIDIA recommends exactly this pattern, a FastAPI WebSocket server, over Triton or Riva, because per-stream state is the bottleneck rather than batching.
The same topology runs locally: on a development machine the inference server and voice backend share one box but still talk over a WebSocket. Moving to production relocates the server to a GPU host behind a secure WebSocket, voice workers run as ordinary CPU pods, and the only application-side change is one environment variable, the server URL. Build the container rather than borrowing it: NVIDIA’s reference Dockerfile.unified compiles from source over two to three hours and bundles a 30B language model and a text-to-speech model an STT deployment never runs, while a purpose-built image shipping only the ASR server and its two dependencies is roughly fifteen lines and builds in minutes.
A ports-and-adapters layout made the model swap clean: the speech-to-text provider is an adapter behind an interface.
- A new adapter that speaks the WebSocket protocol to the inference server.
- A dispatcher that selects the provider from configuration.
- One import line redirected to the dispatcher.
- Two configuration fields: the provider name and the server URL.
The voice pipeline, the business logic, and the conversation flow stayed untouched. The default provider stayed on the cloud API, so nothing broke when the new variable was unset.
One non-obvious detail lives in that adapter: the final transcript has to be ordered ahead of the end-of-speech frame, or Pipecat’s aggregator times out and drops it. A small ordering mistake there loses the last utterance of every turn while every component still reports success.
Weighing whether to move speech-to-text in-house as a voice agent scales? Softcery builds production voice pipelines and the architecture call on which layers to self-host. Schedule a consultation to scope yours.
Latency and accuracy across production GPUs
Speech-to-text on this model sits well under the voice budget on every production GPU. The metric is end-of-speech to final transcript, the finalization the language model waits on. The numbers below combine our own runs with published benchmarks, each row labeled by source.
| GPU | Finalization latency | Concurrent streams | Source |
|---|---|---|---|
| L4 (24 GB) | ~18 ms | ~13, rising to 40–60 at a 320 ms chunk | E2E Networks benchmark; upper figure estimated |
| L40S (48 GB) | under 50 ms | ~96 | Daily.co benchmark (Nemotron 3.5) |
| A100 (80 GB) | ~20–30 ms | ~200–250 | Estimated from the H100 ratio |
| H100 (80 GB) | 24 ms | 560, up to 2,400 at a 1,120 ms chunk | NVIDIA published |
Concurrency scales with chunk size: a larger audio chunk buys more streams per GPU at slightly higher lag, tunable at runtime with no retrain. The latency figures use different methods (per-inference on the L4, utterance time-to-final on the H100), so read the column as sub-100 ms finalization across the board rather than a like-for-like ladder. Our own runs sit alongside these. A dev-box sanity check on a consumer RTX 3060 Ti held an 83 ms median, 105 ms p95, at a single stream; a cloud L4 inside the live pipeline ran 50 to 300 ms end-to-end including the network round-trip, with a 30 to 60 second cold start on fresh model load. The A100 row is the one full estimate, scaled from the H100 figure, and a clean standalone A100 and L40S sweep on our own hardware is the measurement still worth running.
Setup is minimal: the model downloads once as a 2.47 GB file, holds roughly 2 GB of GPU memory, and restarts in 15 to 20 seconds. It picks a latency mode at load, from 80 to 1120 ms of look-ahead, with the 160 ms default the right starting point and mode switching a config value, not a retrain.
Accuracy held where it mattered. Word error rate ran 6.93% to 8.43% depending on latency mode on the English 0.6B, NVIDIA’s published range on LibriSpeech and AMI, on par with top commercial APIs; Nemotron 3.5’s 40-locale average sits near 8.8% WER, English near 7.9%. Short, structured commands of the kind a voice agent actually receives transcribed cleanly. Errors appeared on fast, run-together speech and uncommon words: “crossroads” became “scroll throat,” “unprecedented” came back garbled. Production-grade for an agent fielding short requests.
Cost at saturation: self-hosted vs Deepgram and AssemblyAI
At full GPU saturation, with one GPU serving many concurrent streams and voice-activity detection gating the audio, the self-hosted model lands near $0.00014 per user-minute. Deepgram Nova-3 streaming bills around $0.0048 per minute, AssemblyAI around $0.0025: more than 30 times and roughly 18 times more expensive respectively.
The caveat is saturation. A GPU serving three streams costs the same per hour as one serving three hundred, so the per-minute cost is entirely a function of utilization.
To put a number on the break-even: an L4 runs $0.40 to $0.80 an hour depending on the provider, and Deepgram’s $0.0048 per minute is $0.29 an hour for one continuous stream. Raw compute breaks even near two to three concurrent streams on that L4, and the model serves dozens. Idle capacity, redundancy, and the engineering time to run the server push the practical threshold higher, into the tens of sustained concurrent streams, which is where a GPU stays busy enough to justify the switch. Run the numbers for a specific traffic profile with the voice agent cost calculator before committing either way.
Limits: multilingual routing and untested phone-line audio
Nemotron 3.5 covers 40 locales, but broad coverage is not automatic language handling. Route by the known tenant or merchant locale rather than automatic language detection, which misclassifies short utterances like “yes” or “no” often enough to corrupt the conversation. For a locale 3.5 does not cover, Parakeet TDT v3 adds 25 European languages through a buffered path at slightly higher latency, wired as a second adapter. Our own latency and accuracy numbers came from the English 0.6B predecessor; it shares an architecture and size with 3.5, so they carry over closely, worth confirming on 3.5 directly.
Two pieces of work remain open. The standalone concurrency measurement, noted above, was not finished. Robustness testing across accents, background noise, and telephony codecs was scoped but not run; the model performs well in clean conditions, and how it degrades on real phone-line audio is a question a production deployment must answer.
When self-hosting speech-to-text earns its complexity
Self-hosting adds an inference server, a GPU bill, and an operational surface a cloud API hides. The decision is a threshold, not a preference.
- Concurrent voice traffic is high enough to keep a GPU saturated, where the per-minute economics flip.
- Vendor independence, data residency, or model-version control is a hard requirement.
- The network round-trip to a cloud provider is a measurable part of the latency budget.
- A branded or domain-tuned model is on the roadmap, which a hosted API cannot provide.
When none of those hold, a cloud API is the right answer. The general rule: if self-hosting saves less than 30% after engineering and operational time, the complexity is not worth it. Speech-to-text clears that bar at scale more easily than the rest of the stack, which is why it is the first layer to bring in-house. The language model, covered next in this series, is a harder story.
Softcery builds production voice agents and the infrastructure under them, including the call on which layers to self-host and which to leave on an API. For a specific architecture and the cost model behind it, schedule a consultation.
Frequently Asked Questions
At full GPU saturation, yes, by a wide margin. A self-hosted streaming model on a saturated GPU lands near $0.00014 per user-minute against roughly $0.0048 per minute for Deepgram Nova-3 streaming, more than 30 times cheaper. The catch is saturation: the math only works when one GPU serves many concurrent streams. Below roughly 50 concurrent sessions, a per-minute API is cheaper and simpler.
For short conversational speech, yes. NVIDIA Nemotron Speech Streaming reports 6.93% to 8.43% word error rate depending on latency mode, on par with top commercial APIs and ahead of Whisper for streaming use. In testing, short commands transcribed cleanly; errors appeared on fast, unstructured speech and uncommon words.
On a consumer RTX 3060 Ti we measured a median of 83 ms from end-of-speech to final transcript. NVIDIA reports a 24 ms median on an H100. Both sit well under the sub-300 ms latency a cloud API like Deepgram publishes, because self-hosting removes the network round-trip from every turn.
It scales with GPU and chunk size. Published benchmarks put a single L4 near 13 to 60 streams, an L40S around 96, and an H100 at 560, rising to 2,400 at a 1,120 ms chunk. Concurrency, not single-stream latency, governs how many GPUs a given call volume needs.
Yes. Nemotron 3.5 ASR Streaming covers 40 language locales; the original January 0.6B was English-only. A multilingual deployment routes by the known tenant or merchant locale rather than automatic language detection, which misclassifies short utterances often, and falls back to Parakeet TDT v3 for locales 3.5 does not cover.
Yes. An open source AI voice agent runs all three layers on open weights: speech-to-text is the cleanest to self-host (this article), text-to-speech works when the model clears streaming, batching, and license filters, and the LLM is the hardest, needing quantization and serving tuning. Parts two and three of this series cover the other two layers.
Yes. Open-source AI voice recognition like Nemotron Speech Streaming deploys on-premises on a single GPU, and the WebSocket topology is identical on-prem and in the cloud, so audio never leaves owned infrastructure. That satisfies data-residency requirements no cloud STT API can, and removes the network round-trip from every turn.
