Self-Hosting a Voice AI Stack: Cost, Feasibility, and When It's a Mistake

Self-hosting a voice AI stack works in 2026. For most 30–100 person tech companies it is still the wrong first move.

It becomes right under three conditions:

• Hard data-boundary rule no contract can satisfy.
• Sustained volume high enough to keep owned GPUs busy (roughly 1–2M+ connected minutes/month).
• Voice as core product infrastructure where you need control over latency, reliability, and model behavior that renting cannot give.

Absent one of those, hosted or vendor-managed is cheaper, faster, and more reliable. The numbers here come from costing and prototyping this stack ourselves for a high-traffic e-commerce deployment.

Decide before you size anything

Self-hosting swaps a per-minute bill for a fleet of GPUs, a team to run them, and a pager.

Self-host if most hold:

• A hard driver exists: air-gap or no-egress mandate, a data-residency rule no vendor region meets, a contract banning external AI processors, or volume above roughly 1–2M connected minutes/month.
• You already run GPU inference in production, or will hire the 1.5–2 engineers who can.
• Utilization is high and steady, not a bursty pilot at 10% duty cycle.
• You accept owning model updates, an eval harness, and the 3am incident.

Reconsider when:

• The driver is a vague preference for “control,” yet raw audio still leaves via a hosted carrier.
• Volume is a pilot or seasonal campaign.
• Nobody owns GPU operations.
• A BAA or DPA at each layer already satisfies the requirement.

One more gate: time. Hosted goes live in days. A self-hosted cascade is a months-long project: validate each layer, fine-tune the LLM until tool-calling holds, load-test the fleet. If speed to market is the constraint, the decision is already made.

Latency and escaping shared infrastructure

Response speed is a legitimate reason to move off a chained hosted pipeline, for two reasons that are easy to underrate.

You stop being a tenant on someone else’s busy hour. A public API shares GPUs across every customer. At peak, your p99 rides their queue depth, their rate limits, and their regional load, none of which you control. Dedicated inference, self-hosted or in a vendor VPC, removes that variance. The median may not move much, but the tail gets predictable, and for voice the tail is what the caller hears as an awkward pause.

Colocation removes cross-provider hops. A cascade wired across three separate hosted APIs (STT, LLM, TTS in different clouds) pays a network round-trip and queueing between each layer. Placing all three in one region or VPC collapses that path.

The cheaper first step is a vendor VPC or single-region deployment, which delivers both wins without a self-owned GPU fleet. Full ownership buys the most control over the tail, at the cost of owning that tail yourself. The turn-latency breakdown lives in our latency budget guide.

Reliability cuts both ways

Availability compounds against you in a chained pipeline. Four independent hosted services at 99.9% each multiply to 0.999^4, about 99.6%; at 99% each, to about 96%. Every provider you add is another uncorrelated outage that can take the whole call down, and provider outages are events you cannot control or predict.

Consolidating the stack under one roof reduces the number of independent failure surfaces and lets you engineer fallback end to end, rather than living at the mercy of each vendor’s separate SLA. The catch: a single self-hosted region is itself a single point of failure, and hosted providers give you multi-region redundancy per layer at no extra effort. Owning the stack raises effective uptime only if you actually build the redundancy, health checks, and provider fallback the SLA used to cover. Done well it beats the chained pipeline; done naively it is worse.

Four architectures hide under “self-hosted”

Pick the minimum tier that satisfies the hard constraint. A fully open-weight stack often buys nothing a vendor deployment or a BAA already delivered.

Tier	Inside your boundary	Best for	Burden
Vendor self-host (VPC / on-prem)	Vendor STT/TTS/LLM containers on your infra, vendor quality and support	Data locality over model ownership	Moderate
Hybrid	Audio, STT, redaction on-prem; hosted frontier LLM under BAA on redacted text	Frontier quality where policy permits a processor	Moderate–high
Open-weight full stack	STT, LLM, TTS, orchestration, logs, evals, all owned	High steady volume or hard isolation	High
Air-gapped	No outbound model, license, or telemetry path	Defense, government, contractual isolation	Highest

Vendor self-host is the pragmatic regulated-enterprise option. Deepgram, AssemblyAI, Inworld, Cartesia, and ElevenLabs all ship private-deployment paths. AssemblyAI advertises self-hosted STT at “up to 48 concurrent streams,” on the “same usage-based pricing as our cloud service.” ElevenLabs: “VPC deployments are available now,” with on-prem and on-device “in early access.”

Two traps sit in the vendor path:

• “Self-hosted” rarely means air-gapped. Deepgram’s own docs say a deployment will “only contact the Deepgram license server” and send “metadata such as audio duration or character count.” Content stays local; licensing traffic does not. For a true air gap, confirm it in writing.
• Hardware is locked. Deepgram “only supports NVIDIA GPUs” and states “MIG or partial (aka. fractional) GPUs are not supported,” so you cannot recover economics through GPU sharing.

Media and orchestration are open at every tier. LiveKit is Apache-2.0; Pipecat is BSD 2-Clause. Neither replaces STT, LLM, TTS, evals, or compliance. Barge-in, turn-taking, retries, and tool-call timing live in this layer, and teams routinely under-budget it.

The stack, component by component

The cascade runs media into STT, STT into the LLM, the LLM into TTS, wrapped by orchestration, logging, and evals. Cascading is the only fully self-hostable option today (see real-time vs turn-based). The three models differ sharply in difficulty.

STT: the near-free win

Open-weight streaming recognition matches commercial APIs on the workload voice agents run.

• NVIDIA Nemotron streaming ASR: a “600M parameter model,” runtime chunk sizes of “80ms, 160ms, 560ms, and 1120ms” (the latency-accuracy dial, no retraining), chunk-average “WER (%) 7.67.” At roughly 1.2 GB it delivers high concurrency and ships production-ready.
• A June 2026 multilingual successor extends to “40 language-locales.” Select against that card for multilingual.
• Whisper is a permissive baseline (“MIT License”) but not a streaming layer. You add the wrapper, endpointing, partial-transcript handling, and GPU batching.

Treat 7.67% as a screening ceiling. It is not a production number. Phone audio is 8 kHz narrowband, which clips the detail separating similar consonants and inflates WER. Upsampling cannot recover frequencies above 4 kHz that were never captured. The figure that matters comes from testing on your own call audio, with your accents, codecs, and noise. Full field in our STT/TTS selection guide.

TTS: deceptively hard

Tiny weights, friendly license: Kokoro has “82 million parameters,” an “apache-2.0” license, and ships “8 & 54” languages and voices in v1.0. Then it gets hard.

• Cannot clone a voice. Fixed voicepacks only.
• No native grapheme-to-phoneme. It offloads to espeak-ng (GPL), which raises a copyleft question against the Apache-2.0 model and hands you espeak-ng’s pronunciation errors on names, domain terms, and numbers. There is no built-in SSML or pronunciation-dictionary API to correct them, so you build that layer.
• Sentence-based by default. Most open TTS is not streamable out of the box. Real-time needs sentence-aware chunking, stable prosody while the sentence arrives, and barge-in that cancels synthesis and clears buffers. Commercial TTS ships this; open weights do not.
• Heaviest component on GPU. We model roughly 20–50 streams/GPU (A10G) for TTS against 100–200 for STT, so a fleet needs several times more TTS GPUs. TTS quietly drives the bill.

Costing this for a high-traffic e-commerce deployment, these problems hit even on polished commercial TTS: mispronounced prices and measurements, numbers read as digit strings, punctuation causing wrong pauses, unnatural pacing. The fix was a runtime pronunciation-glossary and text-normalization layer in front of the TTS. A model swap would not have touched it. An open model without the vendor’s built-in normalization needs more of that layer, not less.

Voice choice sets the hardware floor:

• Generic voice: Kokoro is the default.
• Brand voice or cloning: step up to Chatterbox, CosyVoice, or Qwen-TTS, at a bigger footprint and lower concurrency.
• Commercial on-prem raises the floor sharply: Inworld’s on-prem TTS lists a minimum of “GPU NVIDIA H100 SXM5 (80GB).”

LLM: the real cost and the real quality gap

The best open-weight choice for real-time voice is a small-active-parameter mixture-of-experts model. NVIDIA’s Nemotron 3 Nano has “3.5B active parameters and 30B total parameters,” runs on vLLM, and carries an “nvidia-open-model-license.” Its Mamba-2 layers hold a fixed-size recurrent state per session, so memory does not balloon with conversation length the way a dense KV cache does. That is what makes hundreds of concurrent sessions per GPU plausible.

Do not size GPUs from parameter count, because precision and runtime dominate. Lambda shows the BF16 checkpoint at “Weights-on-disk: 58GB” but “Idle VRAM usage: 120GB” on “1x B200 (180 GB SXM6).” The 4-bit NVFP4 variant is far smaller, though runtime, context length, and KV-cache mode set the real footprint. Load the exact checkpoint before budgeting.

The gap is about judgment rather than raw intelligence, and it is real. NVIDIA reports “TauBench V2 (Retail) 54.1” and “TauBench V2 (Airline) 41.5” for Nemotron 3 Nano, which leaves room for domain failures. Voice tool-calling has tighter tolerances than chat: a caller cannot inspect a reasoning trace, and a retry feels awkward.

In our production testing, a base small-MoE model’s tool-calling was not production-ready out of the box. It struggled to decide when to ask a clarifying question versus proceed with a tool call, expected behavior for roughly 3.5B active parameters. Domain fine-tuning was mandatory. Larger open models improve instruction quality but run too slow and costly for real-time. The point of the small fast model is that fine-tuning closes the gap at a fraction of the inference cost. We offer that as a service. For model-versus-model depth, see choosing an LLM for voice agents and LLM selection for AI agents.

GPU capacity and cost

Peak concurrent calls size the fleet. Monthly minutes are the wrong input, because reserved capacity costs the same whether calls arrive or not.

required GPUs = ceil(peak concurrent calls / sessions per GPU) + redundancy
annual GPU cost = hourly GPU price x GPU count x 8760

Sessions per GPU is where planning breaks, and it is latency-bound rather than memory-bound. A GPU can hold hundreds of sessions in VRAM and still blow the turn budget on p95 latency before memory fills, because compute and bandwidth contention spike tail latency as sessions pile on. Google SRE guidance: a high percentile “shows you a plausible worst-case value,” and “latency increases are often a leading indicator of saturation.” Size against p95 and p99 turn latency under load, measured by benchmark.

Our modeled planning numbers, pending a load test on the exact stack. Every figure is a modeled estimate:

Hardware	Workload	Sessions per GPU
B200 homogeneous	all three models	200–400
H100	LLM	40–75
A10G-class	STT	100–200
A10G-class	TTS	20–50

The 200-versus-400 gap per B200 changes annual cost roughly two-fold at high concurrency, which is why a one-week benchmark precedes any hardware commitment.

Two fleet shapes are viable:

• Homogeneous B200 runs all three models on identical nodes: one GPU type, one image, one scaling policy. A DGX B200 carries “1,440 GB total” across “8x NVIDIA Blackwell GPUs,” roughly 180 GB each, ample headroom.
• Heterogeneous H100 + A10G runs separate pools per model, scales each independently, and avoids paying B200 prices for STT and TTS. It is more complex to operate.

GPU rates, checked in 2026 and moving monthly, so re-verify at deployment:

GPU	Rate (per GPU/hr)	Source
B200 SXM6 180GB	$6.69 (8x) to $6.99 (1x)	Lambda
B200	$5.89	RunPod
B200 (p6-b200 on-demand)	~$14.24	AWS, us-east-1
B200 (Capacity Blocks, eff. 2026-07-07)	~$10.30	AWS
H100 SXM 80GB	$4.29	Lambda
A10 24GB	$1.29	Lambda

Procurement carries its own risk. AWS on-demand P6-B200 lists “$113.9328 per On Demand Linux p6-b200.48xlarge Instance Hour,” about $14.24 per GPU-hour, well above specialist GPU clouds. B200 is absent from GCP and the standard Azure catalog, so a homogeneous B200 fleet can mean single-provider dependence on a neo-cloud. Reserve capacity or design for an H100 fallback.

Modeled fleet sizing at three concurrency tiers:

Peak concurrent	B200 homogeneous (200/GPU, N+1)	H100 + A10G
10	2 B200 (redundancy dominates)	1 H100 + 1 A10G STT + 1 A10G TTS
100	2 B200	2–3 H100 + 1 A10G STT + 2–5 A10G TTS
500	4 B200	7–13 H100 + 3–5 A10G STT + 10–25 A10G TTS

One B200 running 24/7 is roughly $52,000/year (RunPod) to $58,600/year (Lambda 8x). A B200 pair looks expensive at 10 concurrent calls because redundancy dominates. It turns rational only at 100-plus steady calls per node, or when isolation is required regardless of cost.

All-in cost, and where the crossover sits

The GPU bill is the visible number. Two others decide it.

Telephony sits in every column. Twilio lists US “Local calls $0.0140 / min” outbound, “$0.0085 / min” inbound, and a “SIP interface $0.0040 / min” for bring-your-own-carrier trunking. That floor of half a cent to a cent and a half per minute applies to the managed platform, the hosted API pipeline, and the self-hosted fleet alike, unless you own SIP trunking and a session border controller.

Staffing moves the decision, and its cost depends entirely on who does the work. A self-hosted stack needs people who run GPU inference in production: vLLM or TensorRT-LLM serving, quantization, KV-cache and batching tuning, autoscaling, model-update pipelines, and the eval harness. Three ways to cover it, at very different cost:

• Hire it net-new. O*NET, on BLS 2025 data, lists national medians of “$135,980” for software developers and “$120,230” for data scientists, before benefits, overhead, or the premium senior GPU-inference talent commands. Load at roughly 1.4x, cover across 1.5–2 heads for on-call, and a dedicated team runs $300,000–500,000 per year. This number kills the economics at low volume.
• Absorb it into an existing team. A CTO or senior engineer who already has the skills and the slack can stand up and run a bounded stack at marginal cost, not a new line item. Realistic for a technical team that treats voice as core infrastructure.
• Bring in a consultancy to design, benchmark, and stand up the stack, then hand it back or operate it, with no permanent headcount. This is the work we do.

The cost tables below assume the expensive net-new path. If an existing team absorbs the work, or a consultancy runs it, the staffing line shrinks and self-hosting turns competitive at far lower volume than the tables imply.

Our modeled all-in monthly comparison. Concurrency is minutes divided by 43,200 per always-busy line per month; peaks run 2–5x the average:

	Small ~50k min (~1–2 avg)	Mid ~500k min (~12 avg)	Large ~3M min (~70 avg)
Managed platform (~$0.08/min + tel.)	~$4.5K	~$45K	~$270K
Self-assembled hosted API (~$0.03–0.05/min)	~$1.5–2.5K	~$15–25K	~$90–150K
Self-hosted inference + telephony	GPUs $2–6K + staff $25–40K → ~$28–46K	GPUs $8–20K + staff → ~$37–64K	GPUs $30–70K + staff → ~$80–135K

Three conclusions follow:

• Below roughly 1M minutes/month, self-hosting loses on the net-new-hire assumption. Hosted wins by two to ten times, because a dedicated team’s cost swamps everything. Absorb the work into an existing team and this crossover moves down sharply.
• Self-hosted inference beats a self-assembled API pipeline on cost only in the multi-million-minute range, and only if the MLOps team already exists for other reasons.
• The managed all-in-one platform is the most expensive at scale and the cheapest to start.

Average concurrency understates the requirement. A 500k-minute month averages about 12 concurrent calls across 24/7, but about 43 if traffic compresses into business hours, before any burst headroom. You size and pay for the peak.

For a hosted anchor, OpenAI’s Realtime pricing lists “gpt-realtime-2 Audio $32.00” per million audio input tokens and “$64.00” output, down from the 2024 launch “$100 per 1M tokens” input. By OpenAI’s own token-duration rule, a balanced call (a minute of user speech, thirty seconds of assistant speech) works out to roughly $0.058/min before text tokens, tools, telephony, and margin. Cost also grows within a session, because “the entire conversation is sent to the model for each Response,” so “turns later in the session will be more expensive.” The full break-even model lives in our analysis of voice agent running costs.

The quality you give up

Quality splits by component. The answer is not uniform.

• STT reaches parity for free. Open-weight streaming matches or beats Whisper on the workload voice agents run. This is a solved problem to self-host.
• TTS reaches audio parity but hands you the frontend. A single generic Kokoro voice sounds competitive. What quality means for a voice agent is broader: correct pronunciation of names, numbers, and prices; stable prosody while streaming; barge-in; and often a specific brand voice. On those, open models trail, because they ship less of the text frontend and streaming layer than a commercial TTS. The voice itself sounds fine. What you inherit is the normalization work the vendor was doing for you.
• The LLM reaches parity only for the domain you fine-tune. A small open model gives you a domain specialist you had to build. A frontier hosted model gives broad competence with zero tuning. Hosted pricing shows this is a quality trade: OpenAI lists “gpt-5.5 $5.00” against “gpt-5.4-nano $0.20” per million input tokens. A cheaper model that fails tool calls or loses conversions can cost more per resolved call than the API fee it saved.

Match the decision to how open-ended the conversation is. A tight, scriptable surface such as intake, ordering, or scheduling loses little to self-hosting. Open-ended conversation loses a lot. Measure quality as business outcome, resolution rate, containment, conversion, and post-call human repair, rather than model-card WER.

When compliance actually forces it

Many buyers assume regulated data mandates on-prem inference. Usually it does not. The normal path is a BAA for HIPAA or a DPA for GDPR at each layer, far cheaper than owning GPUs.

HHS allows cloud PHI processing. A covered entity may use a cloud service for ePHI “provided the covered entity enters into a HIPAA-compliant business associate contract” and “must conduct risk analyses.” The obligation survives encryption: a CSP is a business associate “even if the CSP processes or stores only encrypted ePHI” and “lacks an encryption key.” So “we encrypt before sending” does not remove the contract requirement if the vendor will not sign.

Every layer vendor signs:

• Twilio voice, SIP, and messaging is HIPAA-eligible once you “execute a Business Associate Addendum” on a “Security Edition or Enterprise Edition” plan.
• AssemblyAI has offered a HIPAA BAA since October 2025.
• OpenAI marks its Realtime endpoint zero-data-retention eligible, with limits worth reading: “Web Search with live internet access is not HIPAA eligible,” and “data residency endpoints are charged a 10% uplift” for models released on or after 2026-03-05.

GDPR points the same way. Article 28 requires a “processor providing sufficient guarantees” under a relationship “governed by a contract,” which a DPA and data-residency controls satisfy for many EU-data scenarios. The FTC Safeguards Rule requires that “service providers safeguard customer information,” and banking regulators treat AI vendors as third-party risk to manage rather than infrastructure to forbid.

Private connectivity can substitute for ownership. AWS Bedrock supports “a private connection between your VPC and Amazon Bedrock” with no internet gateway, and states Bedrock “never shares your data with model providers.” That satisfies many “data cannot traverse the public internet” clauses without self-hosting a model.

What actually forces self-hosting is narrow:

• An air-gap or no-egress mandate. A vendor container with a license proxy that phones home may fail it.
• A data-residency or placement requirement no vendor region meets.
• A security organization that will not approve any subprocessor, even where a BAA is legally sufficient.
• Government frameworks such as FedRAMP, IL4/5, or CJIS where the vendor lacks authorization at your level. FedRAMP notes “only a federal agency can determine” whether a use case is in scope.
• A customer contract whose security addendum mandates it.

The theater trap

The most common self-hosting mistake is paying for on-prem inference while the same sensitive data leaves through another door.

If the call rides a hosted carrier, the raw audio, the PHI or PII itself, already left your building at the carrier. Twilio describes PSTN as a “worldwide network of voice-carrying telephone infrastructure,” and a public phone call is not a private path. Self-hosting the models behind a hosted carrier, without the carrier BAA or an owned SIP path, buys nothing on the privacy driver that motivated it.

This inverts the intuitive design. For a privacy driver, the exposure sits in the audio-touching layers, the carrier and STT, and only secondarily in the LLM. The minimum-viable private build is frequently to own the media path, STT, and redaction, and keep the hosted LLM under a BAA if only redacted text reaches it. Map every data class (raw audio, transcripts, tool payloads, recordings, logs) against every component before choosing a tier.

What operating it takes

Self-hosting replaces vendor failover with reliability work you own. A GPU node dying mid-call is your incident, and you build the health checks, autoscaling, provider fallback, and on-call rotation that a hosted SLA covered. As the compounding-availability math above shows, that ownership can raise end-to-end uptime, but only if the redundancy is actually built. Budget for it explicitly.

A production stack needs named owners across distinct functions:

• Voice and app engineering: media pipeline, endpointing, barge-in, orchestration, tool calls, fallback.
• ML inference: model serving, quantization, batching, KV-cache and throughput tuning.
• Platform and SRE: GPU nodes, autoscaling, observability, deploys, on-call, disaster recovery.
• Security and compliance: data flows, contracts, retention, access control, audit evidence.
• Evaluation and QA: call review, synthetic conversation tests, regression suites, release gates.

A vendor-self-hosted deployment can run with 2–3 technical owners plus existing platform support. A full open-weight stack needs 3–5 named owners. They need not all be full-time in steady state, but each function needs real ownership. A 30-person company without GPU and SRE maturity should rarely own the full stack unless voice is strategic product infrastructure.

The recurring costs beyond hardware and salary are easy to miss:

• Idle capacity. You size for a peak of 2–5x sustained and pay for GPUs 24/7. Low average utilization erases the cost advantage.
• Frozen weights. Hosted frontier models improve continuously for free; your weights are frozen until you re-tune and re-validate.
• Evals on every change, or you ship regressions to callers.
• The eval and logging pipeline is itself a data-egress surface. If call logs or evals flow to a hosted tool, that is another BAA point and another leak. See our observability guide.

Start reversible

The decision is not all-or-nothing on day one. Design the cascade with swappable STT, LLM, and TTS behind clean interfaces, so each component can move on-prem incrementally without a rebuild. Start hosted or vendor-self-hosted, prove the product, and migrate the regulated layer later when a hard driver or validated volume justifies it.

Before committing to a GPU or vendor contract, run the one-week benchmark: the exact stack under 10, 50, 100, 200, and 500 synthetic sessions, measuring p95 and p99 turn latency, first-audio and first-token latency, GPU utilization, and error rate, under realistic 8 kHz telephony audio and barge-in rather than happy-path turns. The difference between 100 and 250 sessions per GPU is six figures a year at high concurrency. Guessing is not diligence.

If a hard data-boundary rule, steady high volume, or a strategic model-control need points you toward owning this infrastructure, we run an on-prem feasibility assessment: the data-flow matrix, the tier decision, the fleet-sizing benchmark, and the real cost premium, before you commit budget. That is the work we do at Softcery’s on-premises voice infrastructure practice.