A self-hosted streaming speech-to-text model matched commercial accuracy, beat commercial latency, and cut the per-minute transcription cost by more than 30 times at saturation. One open model, permissive license, fast enough to test on a consumer gaming GPU. This article covers the metric that governs a voice turn, the model that fit, the architecture that scales it to hundreds of concurrent calls, and the numbers we measured against the numbers NVIDIA publishes.

This is the first part of a three-part series on self-hosting the voice stack, alongside the language model and text-to-speech. Speech-to-text comes first because it is the cleanest win: the model is small, the integration is contained, and the results are unambiguous.

The metric that governs the turn: end-of-speech to final transcript

A voice agent feels responsive or sluggish based on one gap: the silence between a caller finishing a sentence and the agent starting to reply. The number that matters is the time from end-of-speech to the final transcript, because the final transcript triggers the language model. Everything downstream waits on it.

Interim transcripts, the partial words that stream in while a person is still talking, gate nothing. They serve live captions and barge-in detection, and their timing follows the physics of articulation rather than server speed. Instrument and optimize the end-of-speech to final number; it sits inside the larger turn-gap budget covered in our voice agent latency budget guide.

Why move speech-to-text off a cloud API

A cloud STT API is the correct first choice for an early-stage voice agent. Three costs accumulate as the product scales:

• Per-minute billing grows linearly with traffic. A voice agent that succeeds turns its own success into a rising variable cost with no ceiling.
• A third-party owns the transcription layer. Provider incidents make the agent go deaf. Pricing, retention, model versions, and data handling sit outside the team's control.
• The network round-trip is baked into every turn. Audio crossing the internet to the provider and back adds 50 to 150 ms per utterance before the model does any work. Self-hosting removes that hop.

None of these matter at ten concurrent sessions. All of them matter at five hundred.

Choosing the model: why Nemotron Speech Streaming beat Riva and Parakeet

Two wrong turns preceded the right model, and both recur in every NVIDIA speech project.

NVIDIA Riva, the official production ASR server, runs older models. New architectures need TensorRT packaging, which lands one to three months behind the open release, and the roughly 10 GB container expects an NGC account plus the NVIDIA container toolkit. Raw NeMo with Parakeet TDT, where the newest models appear first, tops the offline leaderboard but streams through a buffered path with roughly two seconds of right-context latency. Two seconds of dead air after every sentence disqualifies it for a voice agent.

The fit was Nemotron Speech Streaming, released 5 January 2026 and built for real-time voice agents on Pipecat.

The streaming-versus-offline distinction is the whole point. On the Open ASR Leaderboard (word error rate, lower is better), the streaming column separates a voice-agent model from a transcription model:

Nemotron trades a fraction of a point of accuracy against the offline leaders for native low-latency streaming. The correct trade for a conversation.

One model, loaded once: the architecture that scales

Loading a model copy inside every worker process does not survive contact with scale: at 500 concurrent callers, 500 copies at ~1.2 GB each in half precision demand roughly 600 GB of GPU memory, each serving a single stream with no batching.

The architecture that works is a centralized inference server. The model loads once on a GPU host; voice workers stream audio in and transcripts out over a WebSocket. Capacity scales horizontally by adding GPU hosts behind a load balancer. For a model this small, NVIDIA recommends exactly this pattern, a FastAPI WebSocket server, over Triton or Riva, because per-stream state is the bottleneck rather than batching.

The same topology runs locally: on a development machine the inference server and voice backend share one box but still talk over a WebSocket. Moving to production relocates the server to a GPU host behind a secure WebSocket, voice workers run as ordinary CPU pods, and the only application-side change is one environment variable, the server URL. Build the container rather than borrowing it: NVIDIA's reference Dockerfile.unified compiles from source over two to three hours and bundles a 30B language model and a text-to-speech model an STT deployment never runs, while a purpose-built image shipping only the ASR server and its two dependencies is roughly fifteen lines and builds in minutes.

A ports-and-adapters layout made the model swap clean: the speech-to-text provider is an adapter behind an interface.

The adapter hides one subtlety that decides whether transcripts reach the language model at all. Finalization runs in two modes tied to voice-activity detection: a brief pause triggers a soft reset and a fast partial in 30 to 50 ms, sometimes clipped mid-word; true end-of-speech triggers a hard reset that pads roughly 320 ms of trailing silence so the model captures the last word, and only that hard-reset final triggers the language model.

The load-bearing detail is frame ordering. Pipecat's aggregator expects the final transcript before the end-of-speech signal and drops the transcript if a 500 ms timeout fires first. The adapter therefore holds the end-of-speech frame, sends the hard reset, pushes the final transcript, then releases the held frame. Remove that ordering and the pipeline silently loses the last utterance of every turn while every component still reports success. The class of bug that passes a demo and fails a customer.

Weighing whether to move speech-to-text in-house as a voice agent scales? Softcery builds production voice pipelines and the architecture call on which layers to self-host. Schedule a consultation to scope yours.

What we measured: latency, accuracy, and the honest gap

The first real test ran on an RTX 3060 Ti, a consumer gaming card with 8 GB of memory: clone the reference repo, install dependencies, run the server. The model downloads once as a 2.47 GB file and holds roughly 2 GB of GPU memory; restarts reach ready in 15 to 20 seconds. The server picks its latency mode at load, from 80 to 1120 ms of look-ahead. Shorter look-ahead cuts latency at a small accuracy cost; the 160 ms default is the right starting point, and switching modes is a configuration value, not a retrain.

Measured from end-of-speech to final transcript:

The 83 ms median on consumer hardware already beats the sub-300 ms latency Deepgram publishes for streaming. The 24 ms and 560-stream figures are NVIDIA's published H100 benchmarks, not ours; treat them as a ceiling to validate, not a guarantee. On a cloud L4 inside the live pipeline, end-of-speech to final ran 50 to 300 ms including network, with a 30 to 60 second cold start on fresh model load.

Accuracy held where it mattered. Word error rate ran 6.93% to 8.43% depending on latency mode, NVIDIA's published range on LibriSpeech and AMI, on par with top commercial APIs. Short, structured commands of the kind a voice agent actually receives transcribed cleanly. Errors appeared on fast, run-together speech and uncommon words: "crossroads" became "scroll throat," "unprecedented" came back garbled. For a shopping or support agent fielding short requests, production-grade even on the consumer card.

One honest gap: a clean, standalone concurrency sweep on the L4 was not completed. Internal testing pointed to a working ceiling in the low tens of streams per container before tail latency climbs past a 200 ms p95, and that number governs how many GPUs a given traffic level needs. Redo that measurement in isolation before it drives a production capacity model.

Cost at saturation: self-hosted vs Deepgram and AssemblyAI

At full GPU saturation, with one GPU serving many concurrent streams and voice-activity detection gating the audio, the self-hosted model lands near $0.00014 per user-minute. Deepgram Nova-3 streaming bills around $0.0048 per minute, AssemblyAI around $0.0025: more than 30 times and roughly 18 times more expensive respectively.

The caveat is saturation. A GPU serving three streams costs the same per hour as one serving three hundred, so the per-minute cost is entirely a function of utilization. Below roughly 50 concurrent sessions, a per-minute cloud API is both cheaper and far less operational work. The economics turn decisively favorable exactly where the per-minute API bill turns painful. Run the numbers for a specific traffic profile with the voice agent cost calculator before committing either way.

Limits: English-only launch and untested phone-line audio

The English model is English only. NVIDIA's Parakeet TDT v3 covers 25 European languages with buffered streaming, acceptable where slightly higher latency is tolerable. A cleaner option arrived after this work: Nemotron 3.5 ASR Streaming, released in June 2026, brings cache-aware streaming to 40 language locales. Either way, route by the known tenant or merchant locale rather than automatic language detection, which misclassifies short utterances like "yes" or "no" often enough to corrupt the conversation.

Two pieces of work remain open. The standalone concurrency measurement, noted above, was not finished. Robustness testing across accents, background noise, and telephony codecs was scoped but not run; the model performs well in clean conditions, and how it degrades on real phone-line audio is a question a production deployment must answer.

When self-hosting speech-to-text earns its complexity

Self-hosting adds an inference server, a GPU bill, and an operational surface a cloud API hides. The decision is a threshold, not a preference.

When none of those hold, a cloud API is the right answer. The general rule: if self-hosting saves less than 30% after engineering and operational time, the complexity is not worth it. Speech-to-text clears that bar at scale more easily than the rest of the stack, which is why it is the first layer to bring in-house. The language model, covered next in this series, is a harder story.

Softcery builds production voice agents and the infrastructure under them, including the call on which layers to self-host and which to leave on an API. For a specific architecture and the cost model behind it, schedule a consultation.