Self-Hosting a Voice AI Stack: The Short Version

Self-hosting a voice AI stack works in 2026. For most 30–100 person tech companies it is still the wrong first move. It becomes right under three conditions: a hard data-boundary rule no contract can satisfy, sustained volume high enough to keep owned GPUs busy (roughly 1–2M+ connected minutes/month), or voice as core product infrastructure where you need control over latency, reliability, and model behavior that renting cannot give. The numbers here come from costing and prototyping this stack ourselves for a high-traffic e-commerce deployment.

Decide before you size anything

Self-hosting swaps a per-minute bill for a fleet of GPUs, a team to run them, and a pager.

Lean toward it if: a hard driver exists (air-gap, a data-residency rule no vendor region meets, a contract banning external processors, or volume above ~1–2M minutes/month); you already run GPU inference or can absorb it; and utilization is high and steady.

Reconsider if: the driver is a vague preference for “control” while raw audio still leaves via a hosted carrier; volume is a pilot; nobody owns GPU operations; or a BAA at each layer already satisfies the requirement.

One more gate: time. Hosted goes live in days. A self-hosted cascade is a months-long project of validating each layer, fine-tuning the LLM until tool-calling holds, and load-testing the fleet. If speed to market is the constraint, the decision is made.

Latency and reliability are real drivers, but ownership is not the only way to get them:

• Latency. A public API shares GPUs across every customer, so at peak your p99 rides their queue and rate limits. Dedicated inference, self-hosted or in a vendor VPC, removes that variance and tightens the tail the caller hears. Colocating STT, LLM, and TTS in one region also cuts the network hops a cross-cloud pipeline pays.
• Reliability. Availability compounds against a chained pipeline: four hosted services at 99.9% each multiply to about 99.6%, at 99% each to about 96%. Consolidating cuts independent failure surfaces, but a single self-hosted region is its own point of failure.

A vendor VPC deployment delivers most of both wins without a self-owned fleet.

Four architectures hide under “self-hosted”

Pick the minimum tier that satisfies the constraint. A fully open-weight stack often buys nothing a vendor deployment or a BAA already delivered.

Tier	Inside your boundary	Best for	Burden
Vendor self-host (VPC / on-prem)	Vendor STT/TTS/LLM containers on your infra	Data locality over model ownership	Moderate
Hybrid	Audio, STT, redaction on-prem; hosted LLM under BAA on redacted text	Frontier quality where policy permits a processor	Moderate–high
Open-weight full stack	STT, LLM, TTS, orchestration, evals, all owned	High steady volume or hard isolation	High
Air-gapped	No outbound model, license, or telemetry path	Defense, government, contractual isolation	Highest

Deepgram, AssemblyAI, Inworld, Cartesia, and ElevenLabs all ship private-deployment paths, and this is the pragmatic regulated-enterprise option. Two cautions: “self-hosted” rarely means air-gapped (Deepgram’s own docs say a deployment will “only contact the Deepgram license server” and send usage metadata), and hardware is locked to NVIDIA GPUs with no fractional sharing. For media and orchestration, LiveKit (Apache-2.0) and Pipecat (BSD 2-Clause) are open, but neither replaces STT, LLM, TTS, evals, or compliance.

The stack, component by component

Cascading (media into STT into LLM into TTS) is the only fully self-hostable option today. The three models differ sharply in difficulty.

STT: the near-free win

Open-weight streaming recognition matches commercial APIs. NVIDIA’s Nemotron streaming ASR is a 600M-parameter model with a chunk-average 7.67% WER, roughly 1.2 GB, high concurrency, production-ready.

Treat 7.67% as a screening ceiling, not a production number. Phone audio is 8 kHz narrowband, which inflates WER, so the figure that matters comes from testing on your own call audio.

TTS: deceptively hard

Kokoro is 82M parameters under an Apache-2.0 license. Three catches turn it into real work:

• Cannot clone voices. Fixed voicepacks only.
• Pronunciation is outsourced to GPL-licensed espeak-ng, so you inherit its errors on names and numbers, with no SSML API to fix them.
• Sentence-based, not streaming. Real-time audio needs a streaming layer you build.

TTS is also the heaviest speech component on GPU (we model ~20–50 streams/GPU against 100–200 for STT), so it quietly drives the bill. When we costed this on polished commercial TTS, mispronounced prices, digit-string numbers, and wrong pauses still needed a runtime pronunciation-glossary layer. An open model without the vendor’s built-in normalization needs more of that layer, not less.

LLM: the real cost and quality gap

The best open-weight choice for real-time voice is a small-active-parameter mixture-of-experts model. NVIDIA’s Nemotron 3 Nano runs 3.5B active parameters of 30B total on vLLM.

• Do not size GPUs from parameter count. Runtime dominates: the same checkpoint that is 58GB on disk shows 120GB idle VRAM on a B200.
• The gap is judgment. Nemotron 3 Nano scores 54.1 on TauBench V2 Retail, which leaves room for domain failures. In our testing, a base small-MoE model’s tool-calling was not production-ready out of the box.

Domain fine-tuning was mandatory, and we offer it as a service. Larger open models improve quality but run too slow for real-time; the point of the small model is that fine-tuning closes the gap cheaply.

GPU capacity and cost

Peak concurrent calls size the fleet. Monthly minutes are the wrong input, because reserved capacity costs the same idle or busy.

required GPUs = ceil(peak concurrent calls / sessions per GPU) + redundancy

Sessions per GPU is where planning breaks, and it is latency-bound rather than memory-bound: a GPU can hold hundreds of sessions in VRAM and still blow the p95 turn budget before memory fills. Size against p95 and p99 turn latency under load, by benchmark. Our modeled estimates: 200–400 sessions/GPU on a B200 running all three models, 40–75 for an LLM on H100, 100–200 for STT and 20–50 for TTS on A10G-class.

GPU rates, checked 2026 and moving monthly:

GPU	Rate (per GPU/hr)	Source
B200 SXM6 180GB	$5.89–$6.99	RunPod / Lambda
B200 (p6-b200 on-demand)	~$14.24	AWS, us-east-1
H100 SXM 80GB	$4.29	Lambda
A10 24GB	$1.29	Lambda

One B200 running 24/7 is roughly $52,000–58,600/year. Note the procurement risk: AWS on-demand B200 runs ~$14.24/GPU-hr, well above specialist clouds, and B200 is absent from GCP and standard Azure, so a homogeneous B200 fleet can mean single-provider dependence. Reserve capacity or design for an H100 fallback.

All-in cost, and who does the staffing

The GPU bill is the visible number. Telephony sits in every column (Twilio lists US inbound at $0.0085/min, or $0.0040/min for bring-your-own-carrier SIP), whether you self-host or not.

Staffing moves the decision, and its cost depends on who does the work:

• Hire it net-new. A dedicated GPU-inference team (O*NET/BLS 2025 medians of $135,980 for developers and $120,230 for data scientists, loaded ~1.4x across 1.5–2 heads) runs $300,000–500,000/year. This is the number that kills the economics at low volume.
• Absorb it into an existing team. A CTO or senior engineer with the skills and slack can run a bounded stack at marginal cost.
• Bring in a consultancy to design, benchmark, and stand it up, with no permanent headcount. This is the work we do.

Modeled all-in monthly, on the net-new-hire assumption:

	Small ~50k min	Mid ~500k min	Large ~3M min
Managed platform (~$0.08/min)	~$4.5K	~$45K	~$270K
Self-assembled hosted API (~$0.03–0.05/min)	~$1.5–2.5K	~$15–25K	~$90–150K
Self-hosted + telephony	~$28–46K	~$37–64K	~$80–135K

Below roughly 1M minutes/month, self-hosting loses on the net-new-hire assumption, because a dedicated team’s cost swamps everything. Absorb the work into an existing team and that crossover moves down sharply. Self-hosted inference beats a self-assembled API pipeline only in the multi-million-minute range. Size for the peak, not the average: a 500k-minute month averages ~12 concurrent calls across 24/7, but ~43 if traffic compresses into business hours.

The quality you give up

Parity is not uniform across the stack:

• STT reaches parity for free.
• TTS reaches audio parity but hands you the normalization, streaming, and pronunciation-control frontend the vendor was doing.
• The LLM reaches parity only for the domain you fine-tune, never the long tail. A small open model is a specialist you had to build; a frontier hosted model is broad competence with zero tuning. A cheaper model that fails tool calls or loses conversions can cost more per resolved call than the API fee it saved.

Match the choice to how open-ended the conversation is. A scriptable surface (intake, ordering, scheduling) loses little; open-ended conversation loses a lot. Measure quality as business outcome, resolution and conversion, not model-card WER.

When compliance actually forces it

Many buyers assume regulated data mandates on-prem inference. Usually it does not.

• HIPAA allows cloud PHI processing “provided the covered entity enters into a HIPAA-compliant business associate contract.” The obligation survives encryption, so “we encrypt before sending” does not remove it if the vendor will not sign.
• Vendors do sign. Twilio voice/SIP is HIPAA-eligible with a Business Associate Addendum; AssemblyAI has offered a BAA since October 2025.
• GDPR Article 28 is satisfied by a DPA and data-residency controls. AWS Bedrock offers a private VPC connection with no internet gateway that meets many “no public internet” clauses without self-hosting.

What actually forces ownership is narrow: an air-gap or no-egress mandate, a data-residency rule no vendor region meets, a security org that will not approve any subprocessor, government frameworks (FedRAMP, IL4/5, CJIS) where the vendor lacks authorization, or a customer contract that mandates it.

The theater trap

The most common mistake is paying for on-prem inference while the same data leaves through another door. If the call rides a hosted carrier, the raw audio (the PHI or PII itself) already left at the carrier.

For a privacy driver, exposure sits in the audio-touching layers, the carrier and STT, and only secondarily in the LLM. The minimum-viable private build is often to own the media path, STT, and redaction, and keep the hosted LLM under a BAA if only redacted text reaches it. Map every data class against every component before choosing a tier.

Start reversible

Design the cascade with swappable STT, LLM, and TTS behind clean interfaces, so each component can move on-prem incrementally. Start hosted or vendor-self-hosted, prove the product, and migrate the regulated layer later when a hard driver or validated volume justifies it. Before committing to any GPU or vendor contract, run a one-week benchmark of the exact stack under 10, 50, 100, 200, and 500 synthetic sessions, measuring p95/p99 turn latency and error rate under realistic 8 kHz audio and barge-in. The difference between 100 and 250 sessions per GPU is six figures a year at high concurrency.

If a hard data-boundary rule, steady high volume, or a strategic model-control need points you toward owning this, we run an on-prem feasibility assessment: the data-flow matrix, the tier decision, the fleet-sizing benchmark, and the real cost premium, before you commit budget. That is the work we do at Softcery’s on-premises voice infrastructure practice.