Our website copilot Waffles uses Cartesia Ink-2 STT. On 4 live calls it took 1.1 s from the end of speech to the first byte of text. That is 59 % of the 1.9 s the caller waits before the first audio comes back, the gap our voice agent latency budget splits. The LLM and the TTS on rented 3090s take 0.4 and 0.3 s of the rest. The LLM is Gemma 4 26B-A4B on a rented RTX 3090. The recognizer is the slow part of the chain.

NVIDIA ships Nemotron ASR Streaming, an English streaming recognizer at 0.6B parameters, and sherpa-onnx ships it as an int8 export that runs on a CPU. We put it on a 6-core desktop and measured how many callers it serves and how long the final transcript takes. Self-hosting streaming STT measured the same end-of-speech to final transcript number on a GPU.

The outcome

On a desktop with a Ryzen 7500F, 6 cores and 32 GB, in a container, no card. The export is the sherpa-onnx int8 asset of nvidia/nemotron-speech-streaming-en-0.6b for the 1120 ms chunk, 464 MB, on sherpa-onnx 1.13.8 and onnxruntime 1.28.2:

sherpa_onnx.OnlineRecognizer.from_transducer(
    tokens="tokens.txt", encoder="encoder.int8.onnx",
    decoder="decoder.int8.onnx", joiner="joiner.int8.onnx",
    num_threads=6, sample_rate=16000, feature_dim=128,
    enable_endpoint_detection=False, decoding_method="greedy_search",
    blank_penalty=1.0, provider="cpu:session.conf")

session.conf holds one line, session.intra_op.spin_duration_us=200. The served config is the 1120 ms chunk, 6 threads per stream and 0.5 s of preroll. One socket per call, one recognizer stream per turn.

Two measures repeat below. The final latency is the time from the end of the turn to the final transcript. The real time factor, RTF, is processing time over audio time. At speech speed the client sends audio as fast as it is spoken. Unpaced, it sends with no wait, so the RTF is the processing cost per stream.

The bench feeds each clip over the socket in 20 ms frames at speech speed, then ends the turn, on the host that serves, over loopback. The 24 clip set holds the p50 within 2.5 ms of the 300 clip set and understates the p90 by 27 ms.

1. The chunk size

The encoder is cache-aware. It takes one chunk of audio, reads 90 ms past it for right context, and carries its state to the next chunk. The export bakes the chunk in, so one chunk size is one asset of 464 MB and a change of chunk is a new pull. The English release ships 80, 160, 560 and 1120 ms. The 40-language release, 3.5, ships those and 320 ms.

The chunk sets 3 things. The client sees no partial until the first chunk fills. Finalization, the last encoder call after the turn ends, runs over one chunk. And the processing time per second of audio rises as the chunk falls. In process, 4 threads, the 16.71 s test clip of the export, medians of 3 to 5 repetitions:

Finalization is one encoder call. The chunk times the processing time per millisecond predicts it within 7 %. It has a floor of 45 ms, and the 80 ms export cuts 0.1 ms off the 160 ms export for 2.4 times the processing. From 1120 ms down to 80 ms the processing per second of audio rises 10.5 times.

The English model beats the 3.5 model by about one point at every shared chunk, on this clip and on the published rows. The 3.5 release holds the only 320 ms export, and its word error rate there is worse than the English 160 ms export.

The same finalization reads 73.8 ms in process and 73 ms over the websocket, on one clip.

Another process held 1 to 2 cores for part of this run, so the rows give the shape. Section 7 re-measures 560 against 1120 over the socket on 300 clips.

2. The turn

The client sends a start-of-turn message, then signed 16-bit mono at 16 kHz in 20 ms frames, then an end-of-turn message. The endpoint answers each partial transcript and one final per turn. The client sets the turn boundary. The recognizer has an endpoint detector and we keep it off, because the agent already decides when the caller has stopped.

One recognizer stream serves one turn. The encoder carries state from chunk to chunk, so a stream cut inside an utterance drops the left context, and a cut inside a word loses the last words. A cut at an utterance end loses no words. The rule for the agent: cut at an utterance end.

The voice activity detector on the agent reports speech late. So audio flows for the whole call, and the endpoint keeps the last 0.5 s before the start of the turn and feeds it into the new stream first. The check: a turn that opens 500 ms late keeps its first word.

At the end of the turn the endpoint feeds one chunk of silence and then ends the input. Without the padding the last partial chunk never reaches the encoder and the final loses its last words. One chunk of padding against four on 40 clips loses 0 words at every chunk. The text differs on 0 clips at 1120 ms, 1 at 560 and 2 at 160. The 90 ms right context of the export does not change the final. An end of turn with no speech answers with an empty final, at every caller count.

The endpoint takes a bearer key, and a connection without the key closes with code 1008. There is no TLS. The key and the audio cross the network in the clear.

3. Threads

The thread count is the onnxruntime intra-op count of one stream, the threads one encoder call spreads over. Each connection runs its decode in its own worker thread, so 12 callers at 6 threads ask for 72 threads on 12 hardware threads. 1120 ms chunk, 24 clips, no session option, one repetition per point:

The final latency falls with the thread count and flattens: 196, 114, 86, 71 and 61.5 ms from 1 to 6 threads. It stops falling at 6, the core count. 8 and 12 threads lose at 1 caller, and at 12 callers their p90 reads 316 and 422 ms against 237. The oversubscription at 6 threads and 12 callers does not invert the order.

The table ranks latency and hides the price of a thread. Two runs of one config differ by 2.5 ms at 1 caller and by 16.5 ms at 12, so a 1 ms gap in the table is noise. On the mac, 2 threads cost half the processor of 4 at 1 caller for the same wall time, section 8. We run 6 on the desktop.

4. The runtime session file

An onnxruntime thread pool spins after a task, waiting for the next one, before it blocks. By default the spin runs a fixed iteration count, and its length in time follows the CPU. spin_duration_us sets the length in time instead, and the onnxruntime 1.28.2 session option header names 500 to 2,000 us for client workloads. The settings are session options, and sherpa-onnx gives no argument for them. Its provider string takes a path after a colon, cpu:session.conf, and reads one key per line from the file. A key under the SessionConfig prefix reaches onnxruntime. An unknown key drops without a message. The positive control is ProfilingFilePrefix: it writes one profile per graph, and 3 files appear, for the encoder, the decoder and the joiner.

1120 ms chunk, 4 threads, 12 callers, 24 clips, one repetition per row:

Spin off and spin stop cut the processing 6 % and add 13 to 32 ms to the final latency. The 2 block sizes add 3 and 17 ms and cut no processing. A 200 us spin cuts the processing and keeps the latency, so we repeated it, one row per run:

The option and the thread count interact. At 4 threads it cuts the processing 8 %. At 6 threads it cuts 25 %, 0.391 to 0.295 at 12 callers. We serve 6 threads and the option. Against 4 threads and no option, that cuts the final latency 21 % at 1 caller, 73.5 to 58 ms, and the processing 21 %, 0.085 to 0.067.

5. Callers per host

One recognizer serves every connection, and each connection decodes in its own thread. The cost comes with the streams that hold a full chunk at the same time, because each one calls the encoder. The bench refuses a run with a 1 minute load over 1.0. The other containers on the host sit idle under 1.3 % of one core.

1120 ms chunk, 4 threads, one repetition per point. 24 clips up to 12 callers and 48 clips from 12, so every point runs full batches:

The recognizer keeps up with speech to 24 callers. The RTF at speech speed stays at 1.01 to 1.03, and the breaking point sits between 24 and 48. Unpaced throughput rises 2.95 times from 1 caller to 12, 11.8 to 34.8 times speech speed at 4 threads, and 14.9 to 36.7 at 6 threads with the spin option.

Tail latency sets the limit. The p90 rises from 93 ms at 1 caller to 619 ms at 24, while the p50 moves 106 ms. So the ceiling depends on the limit the operator sets. At 114 ms p90 it is 6 callers. At 250 ms it is 12. At 1,125 ms, the rented mean, it is 24. We set 250 ms and serve 12.

Load does not change the text. The word error rate reads 3.4 mean at every caller count of the 24 clip set. It reads 2.3 on 48 clips and 3.1 on 300, so a small corpus does not rank text, and the sweep rows rank latency only.

The served config, 6 threads and spin, on 300 clips at 3 and at 12 callers and 48 clips at 24: 62 and 74 ms at 3, 66 and 212 at 12, 125 and 575 at 24. It serves 12 callers at the limit and breaks at 24.

6. The batched decode

The runtime offers one decode call over many streams. We moved the decode to one thread over the ready streams and dropped the thread per connection. Spin 200 us on every row:

The batched call loses. Audio arrives in 20 ms frames, so at any instant few streams hold a full chunk. The one thread then runs the streams in series where the worker threads ran them side by side. At 24 callers the RTF at speech speed passes 1, at 1.182, and the final takes a second. The batched entry also couples the streams: one slow stream holds the whole batch. The endpoint keeps one thread per connection.

7. The 560 ms chunk

The same config with the 560 ms export halves the wait for the first chunk and doubles the encoder calls. 300 clips at 12 callers, 6 threads and spin:

The shorter chunk cuts the final latency 12 ms and costs 46 % more processing per call. Both serve 12 callers at the 250 ms limit. At 24 the p90 reads 1,033 ms against 575. The chunk does not change the text on this set. Both read 3.1 mean and 0.0 median.

The first partial moves 159 ms, a quarter of the 560 ms the chunk moves. The first partial waits for the first spoken word, and the clips carry leading silence. The clock starts at the first audio frame, 500 ms before the start of the turn, so the preroll sits inside it. At 1 caller on the test clip the 1120 ms chunk gives 1,301 ms.

A deployment that shows partials during a turn pays the processing for them. Ours shows the final only, so we serve 1120 ms. We keep no 160 ms config: 560 ms already costs 46 % more processing for the same text.

8. Memory and cost per audio second

The desktop rows rank latency and hold no processor cost. We sampled the cost on a mac, in process, no socket and no container. Apple M2, 1120 ms chunk, no session option, the 2 test clips of the export, 3 repetitions per caller:

The process uses 33 MB before the load and 977 to 1,136 MB after it. 4 streams add about 100 MB. 2 threads cost half the processor of 4 at 1 caller, 0.176 against 0.364 core-seconds per audio second, for a final latency of 85 against 79 ms. The cost per audio second stays flat from 1 to 3 callers and rises 17 % at 4. 4 callers at 2 threads take 0.83 cores while every caller speaks, and the final latency doubles from 85 to 165 ms, still 6 times under the rented mean.

The deployment target is the host the agent runs on, a Hetzner cx23 at 5.49 euro a month, 2 shared vCPU and 4 GB. The agent caps calls at 4. A shared Intel vCPU runs slower than an M2 performance core and loses cycles to its neighbours, so each mac row is a floor. The next tier is a cx33, 4 vCPU and 8 GB, at 8.49 euro.

9. What we did not measure

• A shared vCPU. The cost rows come from an M2.
• A price per voice minute.
• The blank penalty and the preroll. Every row uses 1.0 and 0.5 s.
• The 160 ms and 80 ms English exports over the socket. The chunk table runs the 3.5 release in process.
• A gating corpus. Every clip is LibriSpeech test-clean read speech at 16 kHz, no browser channel.
• A word error rate under 300 clips. 24 clips read 3.4 and 48 read 2.3 on one config.
• The published normalizer. NVIDIA reports 2.32 % on LibriSpeech test-clean at 1120 ms with whisper-normalizer. Ours drops case and punctuation only, so 3.1 % and 2.32 % are not one measure.
• Beam search. The runtime ships greedy only for this transducer.
• Another sample rate. The endpoint skips a clip at another rate and resamples nothing.
• The resident memory and the idle processor of the agent host under a call.
• A second runtime provider. xnnpack falls back to the CPU without a message.
• A second host for the latency rows. Each number is one Ryzen 7500F, 1 repetition per sweep point and 300 clips per served row.