Self-Hosting a Voice AI Stack: Cost, Feasibility, and When It's a Mistake

Self-hosting a voice AI stack is viable in 2026, and for most 30–100 person technology companies it is still the wrong first move. It becomes the right one under three conditions: a hard data-boundary rule that no contract can satisfy, sustained call volume high enough to keep owned GPUs busy, or voice as core product infrastructure that needs model control you cannot rent.

Absent one of those, a hosted or vendor-managed path is cheaper, faster, and more reliable.

That conclusion comes from costing and prototyping this stack ourselves, for a high-traffic e-commerce deployment. The GPU math, the quality gaps, and the failure modes below are measured against that work, not borrowed.

Decide before you size anything

Self-hosting swaps a per-minute bill for a fleet of GPUs, a team to run them, and a pager. That trade pays off narrowly. Before pricing hardware, run both checklists.

Self-host if you can say yes to most of these:

• A hard driver exists: an air-gap or no-egress mandate, a data-residency rule no vendor region meets, a customer or government contract that bans external AI processors, or sustained volume above roughly 1–2 million connected minutes per month.
• You already operate GPU inference in production, or you will hire the 1.5–2 engineers who can.
• Utilization will be high and steady, not a bursty pilot running at a 10% duty cycle.
• You accept owning model updates, an evaluation harness, and the 3am incident.

It is a mistake when:

• The driver is a vague preference for “control” or “privacy,” but the raw call audio still leaves through a hosted telephony carrier.
• Volume is a pilot or a seasonal campaign.
• Nobody on the team owns GPU operations.
• A business associate agreement (BAA) or data processing agreement (DPA) at each layer already satisfies the requirement.

Timeline is the last gate before any cost model. A hosted voice agent goes live in days. A self-hosted cascade is a project: standing up and validating each pipeline layer, fine-tuning the language model until its tool-calling holds, then load-testing the fleet. Realistic time to first production is months. If speed to market is the constraint, the decision is already made.

Four architectures hide under “self-hosted”

Self-hosting is not a single switch. It is a spectrum of four tiers, and the right move is the minimum tier that satisfies the hard constraint. A fully open-weight stack often buys nothing that a vendor-managed deployment or a BAA already delivered.

Tier	What sits inside your boundary	Best for	Operating burden
Vendor self-host (VPC / on-prem)	Vendor STT/TTS/LLM containers on your infrastructure, vendor model quality and support	Regulated teams that need data locality more than model ownership	Moderate
Hybrid	Audio, STT, and redaction on-prem; hosted frontier LLM under BAA/DPA on redacted text	Teams that need frontier LLM quality where policy permits an approved processor	Moderate to high
Open-weight full stack	STT, LLM, TTS, orchestration, logs, evals, monitoring, all owned	High steady volume or hard isolation	High
Air-gapped	No outbound model, license, or telemetry path; offline artifact supply chain	Defense, some government, contractual isolation	Highest

Vendor self-host is the pragmatic regulated-enterprise option. Deepgram, AssemblyAI, Inworld, Cartesia, and ElevenLabs all ship private-deployment paths.

AssemblyAI advertises self-hosted STT at up to 48 concurrent streams on your infrastructure, at the same usage-based pricing as its cloud service. ElevenLabs offers VPC deployments now, with on-premise and on-device in early access.

Vendor “self-hosted” rarely means air-gapped. Deepgram’s own docs describe a self-hosted deployment that will “only contact the Deepgram license server” and send “metadata such as audio duration or character count.” Request content stays local; licensing and usage traffic does not. If the driver is a true air gap, confirm it in writing, not from a marketing label.

Vendor deployments also carry hardware constraints. Deepgram supports only NVIDIA GPUs and does not support MIG or fractional GPUs, so you cannot claw back economics through GPU sharing.

For the media and orchestration layer, open-source infrastructure exists at every tier. LiveKit’s server is Apache-2.0; Pipecat, a framework for voice and multimodal agents, is BSD-2-Clause.

Neither replaces STT, LLM, TTS, evaluation, or compliance work. Barge-in, turn-taking, retries, and tool-call timing live in this layer, and teams routinely under-budget it.

The stack, component by component

A self-hosted cascade runs media transport into STT, STT into the LLM, the LLM into TTS, wrapped by orchestration, logging, and evaluation. For the cascading-versus-speech-to-speech choice, see our real-time vs turn-based architecture guide; cascading is the only fully self-hostable option today.

The three models carry sharply different difficulty. STT is a near-free win. TTS looks trivial and is not. The LLM is where the real cost and the real quality gap concentrate.

STT: self-host it without drama

Open-weight streaming recognition now matches commercial APIs on the workload voice agents actually run. NVIDIA’s Nemotron streaming ASR is a 600M-parameter model with runtime chunk sizes of 80, 160, 560, and 1120 ms, the dial that trades latency for accuracy without retraining. NVIDIA reports a 7.67% chunk-average WER across its evaluated benchmarks.

At roughly 1.2 GB it delivers high concurrency per GPU and ships production-ready. NVIDIA also notes a June 2026 multilingual successor extending the stack to 40 language-locales, so multilingual deployments should evaluate that model, not the English-only checkpoint.

Treat 7.67% as a screening number and a best case, not a production number. Phone agents run 8 kHz narrowband audio, which clips the high-frequency detail that separates similar consonants and inflates word error rate. Upsampling does not recover it; the frequencies above 4 kHz were never captured. The accuracy figure that matters comes from testing on your own call audio, with your accents, codecs, and noise, not from a leaderboard.

Whisper remains a permissively licensed baseline: MIT-licensed and general-purpose. But the base project is not a streaming voice-agent layer. Live phone agents need the streaming wrapper, endpointing, partial-transcript handling, and GPU batching built around it.

For the full STT and TTS selection field, including commercial APIs, see our STT/TTS selection guide.

TTS: the deceptively hard component

The weights are tiny and the license is friendly. Kokoro has 82 million parameters, an Apache-2.0 license, and ships 8 languages and 54 voices in v1.0. That is where the easy part ends.

Kokoro cannot clone a voice; it ships fixed voicepacks only. It also does not do its own grapheme-to-phoneme conversion. It offloads that to espeak-ng, a separate GPL-licensed tool, which raises a copyleft question against the otherwise Apache-2.0 model and, more practically, hands you espeak-ng’s pronunciation errors on names, domain terms, and numbers.

There is no built-in SSML or pronunciation-dictionary API to correct them. You build that layer.

Most open TTS models are sentence-based and are not streamable by default. Real-time voice needs an engineered streaming layer: sentence-aware chunking, stable prosody while the sentence is still arriving, and barge-in that cancels in-progress synthesis and clears audio buffers. A commercial streaming TTS ships this. Open weights do not.

TTS is the heaviest speech component on GPU. We model roughly 20–50 concurrent streams per A10G-class GPU for TTS against 100–200 for STT, so a fleet needs several times more TTS GPUs than STT GPUs. TTS quietly drives the GPU bill in a self-hosted cascade.

When we costed self-hosted voice for a high-traffic e-commerce deployment, these problems appeared even on a polished commercial TTS: mispronounced prices and measurements, numbers read as digit strings, punctuation producing wrong pauses, and unnatural pacing.

The fix was a runtime pronunciation-glossary and text-normalization layer in front of the TTS, covering global and per-domain terms, currency, and units. A model swap would not have touched it.

If a managed vendor’s TTS still needs that layer for a real domain, an open model that lacks the vendor’s built-in normalization needs more of it, not less.

If a single generic voice is acceptable, Kokoro is the default. If you need a brand voice or cloning, step up to a larger model such as Chatterbox, CosyVoice, or Qwen-TTS, at a bigger footprint and lower concurrency per GPU. Commercial on-prem TTS raises the hardware floor sharply: Inworld’s on-prem TTS lists an 80 GB H100 as the minimum GPU.

LLM: the real cost and the real quality gap

The open-weight shape that fits real-time voice is a small-active-parameter mixture-of-experts model. NVIDIA’s Nemotron 3 Nano has 3.5B active parameters of 30B total, runs on vLLM, and ships under the NVIDIA Open Model License.

The architecture makes hundreds of concurrent sessions per GPU plausible: the Mamba-2 layers hold a fixed-size recurrent state per session, so per-session memory does not balloon with conversation length the way a dense transformer’s KV cache does.

Do not size GPUs from parameter count. Precision and runtime dominate. Lambda’s deployment guide shows the BF16 checkpoint at 58 GB on disk but 120 GB of idle VRAM on a single 180 GB B200. The 4-bit NVFP4 variant is far smaller, but the runtime, context length, and KV-cache mode set the real footprint. Load the exact checkpoint before you budget.

The quality gap here is about judgment, not raw intelligence, and it is real. NVIDIA reports Nemotron 3 Nano agentic scores of 54.1 on TauBench V2 Retail and 41.5 on Airline, which leave room for domain failures. Voice tool-calling has tighter tolerances than chat, because a caller cannot inspect a reasoning trace and a retry feels awkward.

In our production testing, a base small-MoE model’s tool-calling and conversational judgment were not production-ready out of the box. It struggled to decide when to ask a clarifying question versus proceed with a tool call, expected behavior for a model with roughly 3.5B active parameters. Domain fine-tuning was a requirement, not an optimization.

Larger open models improve judgment and instruction-following but are too slow and costly for real-time voice. The entire point of the small fast model is that fine-tuning closes the gap at a fraction of the inference cost.

That fine-tuning is a program with data, evaluations, and an owner; we offer it as a service. For model-versus-model selection depth, see our guides on choosing an LLM for voice agents and LLM selection for AI agents.

GPU capacity and what it costs

Peak concurrent calls size the fleet, not monthly minutes. Reserved GPU capacity costs the same whether calls arrive or not, so the number that matters is how many simultaneous sessions you must serve at your busy hour, plus redundancy.

The sizing method:

required GPUs = ceil(peak concurrent calls / sessions per GPU) + redundancy
annual GPU cost = hourly GPU price x GPU count x 8760

The load-bearing input, sessions per GPU, is where planning goes wrong. It is a latency-bound number, not a memory-fit number. A GPU may hold hundreds of sessions in VRAM and still blow the turn budget on p95 latency well before memory fills, because compute and memory-bandwidth contention spike tail latency as sessions pile on.

Google’s SRE guidance is blunt on why the tail matters: a high-order percentile “shows you a plausible worst-case value,” and “latency increases are often a leading indicator of saturation.” Size against p95 and p99 end-to-end turn latency under concurrent load, measured by a benchmark rather than a memory calculation.

Our modeled planning numbers, pending a load test on the exact stack, are:

• B200 homogeneous (all three models per GPU): 200–400 concurrent sessions per GPU.
• H100 for the LLM: 40–75 sessions per GPU.
• A10G-class for STT: 100–200 streams per GPU.
• A10G-class for TTS: 20–50 streams per GPU.

The gap between 200 and 400 sessions per B200 changes annual cost roughly two-fold at high concurrency, which is why a one-week benchmark on the real STT, LLM, TTS, endpointing, and tool-call pattern precedes any hardware commitment.

Two fleet shapes are viable.

Homogeneous B200. All three models run on identical nodes: one GPU type, one image, one scaling policy, fewer cross-service hops. A DGX B200 node carries 1,440 GB across eight Blackwell GPUs, roughly 180 GB per GPU, ample to hold the stack with headroom.

Heterogeneous H100 plus A10G. Separate pools per model, scaled independently, no B200 prices paid for STT and TTS. More complex to operate.

GPU rates move monthly. These were checked mid-2026; re-verify at deployment:

GPU	Rate (per GPU/hr)	Source
B200 SXM6 180GB	$6.69 (8x) to $6.99 (1x)	Lambda
B200	$5.89	RunPod
B200 (p6-b200 on-demand)	about $14.24	AWS EC2 Price List, us-east-1
B200 (Capacity Blocks, effective 2026-07-07)	about $10.30	AWS
H100 SXM 80GB	$4.29	Lambda
A10 24GB	$1.29	Lambda

Procurement matters. AWS lists on-demand p6-b200.48xlarge at $113.93 per instance-hour, about $14.24 per GPU-hour, materially above specialist GPU clouds. B200 is also absent from GCP and the standard Azure catalog, so committing to a homogeneous B200 fleet can mean single-provider dependence on a neo-cloud. Reserve capacity or design for an H100 fallback.

Modeled fleet sizing at three generic concurrency tiers:

Peak concurrent	B200 homogeneous (200/GPU, N+1)	H100 + A10G heterogeneous
10 sessions	2 B200 (redundancy dominates)	1 H100 + 1 A10G STT + 1 A10G TTS
100 sessions	2 B200	2–3 H100 + 1 A10G STT + 2–5 A10G TTS
500 sessions	4 B200	7–13 H100 + 3–5 A10G STT + 10–25 A10G TTS

At the RunPod B200 rate, one GPU running 24/7 is roughly $52,000 per year; at Lambda’s 8x rate, roughly $58,600. A B200 pair looks expensive at 10 concurrent calls because redundancy dominates. It grows rational only when the same always-on node serves 100-plus steady calls, or when the buyer needs the isolation regardless of cost.

The all-in cost, and where the crossover really sits

The GPU bill is the visible number. The decision turns on two others.

Telephony sits in every column, and cost comparisons routinely omit it. Twilio lists US local calls at $0.014 per minute outbound and $0.0085 inbound, with a SIP interface at $0.004 per minute for bring-your-own-carrier trunking. That floor of half a cent to a cent and a half per minute applies to the managed platform, the hosted API pipeline, and the self-hosted fleet alike, unless you also own SIP trunking and a session border controller.

Staffing flips the decision. Operating a self-hosted voice stack needs people who run GPU inference in production: vLLM or TensorRT-LLM serving, quantization, KV-cache and batching tuning, autoscaling against a bursty concurrency curve, model-update pipelines, and the evaluation harness.

Wage data sets the floor. O*NET, using Bureau of Labor Statistics 2025 wage data, lists a national median annual wage of $135,980 for software developers and $120,230 for data scientists, before benefits, overhead, or the premium senior GPU-inference talent commands. Load that at roughly 1.4x, add 1.5–2 heads of coverage so one person is not the entire on-call rotation, and the fixed cost lands at $300,000–500,000 per year, independent of call volume.

Our modeled all-in monthly comparison across three volume tiers. Concurrency derives from monthly minutes divided by 43,200, the minutes one always-busy line carries per month; peaks run 2–5x the 24/7 average:

	Small, ~50k min/mo (~1–2 avg concurrency)	Mid, ~500k min/mo (~12 avg)	Large, ~3M min/mo (~70 avg)
Managed platform (all-in ~$0.08/min + telephony)	~$4.5K/mo	~$45K/mo	~$270K/mo
Self-assembled hosted API pipeline (STT+LLM+TTS+telephony ~$0.03–0.05/min)	~$1.5–2.5K/mo	~$15–25K/mo	~$90–150K/mo
Self-hosted inference + telephony	1–2 GPUs mostly idle ($2–6K) + staffing ($25–40K) → ~$28–46K/mo	3–8 GPUs ($8–20K) + staffing ($25–40K) → ~$37–64K/mo	10–30 GPUs ($30–70K) + staffing ($25–40K) → ~$80–135K/mo

The pattern is the point. Below roughly 1 million minutes per month, self-hosting is not close: hosted wins by two to ten times once staffing is counted, because the fixed team cost swamps everything.

Self-hosting inference beats a self-assembled hosted API pipeline on cost only in the multi-million-minute range, and only if the MLOps team already exists for other reasons. The managed all-in-one platform is the most expensive at scale and the cheapest to start.

Average concurrency also understates the real requirement. A 500,000-minute month averages about 12 concurrent calls across 24/7, but roughly 43 if the traffic compresses into business hours, before any burst headroom. You size and pay for the peak.

For a hosted comparison anchor, OpenAI’s current Realtime pricing lists gpt-realtime-2 audio at $32 per million input tokens and $64 per million output, down from $100 per million input at the 2024 launch. Applying OpenAI’s own token-duration rule, a balanced call with a minute of user speech and thirty seconds of assistant speech works out to roughly $0.058 per minute before text tokens, tools, telephony, and platform margin.

Realtime cost also grows across a session: the entire conversation is resent to the model on each response, so later turns cost more. The full break-even model across managed, bring-your-own, and self-assembled paths lives in our analysis of voice agent running costs.

The quality you give up

Quality splits by component.

STT: parity. Open-weight streaming recognition matches or beats Whisper on the streaming workload voice agents run. A solved problem.

TTS: audio parity, but the frontend is yours. A single generic Kokoro voice sounds competitive. Voice-agent quality is broader, though: correct pronunciation of names, numbers, and prices; stable prosody while streaming; barge-in; often a specific brand voice. Open models trail there because they ship less of the text frontend and streaming layer. You inherit the normalization and pronunciation-control work the vendor was doing.

LLM: parity only in the domain you fine-tune. A small open model is a domain specialist you had to build; a frontier hosted model is broad competence with zero tuning. The hosted price spread shows this is a quality trade, not only a hosting trade: OpenAI lists gpt-5.5 at $5.00 per million input tokens against gpt-5.4-nano at $0.20. A model that fails tool calls or loses conversions costs more per resolved call than the API fee it saved.

Match the decision to how open-ended the conversation is. Tight, scriptable tasks – intake, ordering, scheduling – lose little to self-hosting. Open-ended conversation loses a lot. Measure quality as business outcomes – resolution rate, containment, conversion, post-call human repair – not model-card WER.

When compliance actually forces it

Many buyers assume regulated data mandates on-prem inference. Usually it does not. The regulated path is normally a BAA for HIPAA or a DPA for GDPR at each layer, far cheaper than owning GPUs.

HIPAA allows cloud processing. HHS is explicit: a covered entity may use a cloud service to process ePHI “provided the covered entity enters into a HIPAA-compliant business associate contract” and “must conduct risk analyses.”

The obligation survives encryption: a cloud provider is a business associate “even if the CSP processes or stores only encrypted ePHI” and “lacks an encryption key.” So “we encrypt before sending to the model” does not remove the contract requirement if the vendor will not sign.

Every layer vendor signs. Twilio’s voice, SIP, and messaging products are HIPAA-eligible on Security Edition and Enterprise Edition plans once a Business Associate Addendum is executed. AssemblyAI has offered a HIPAA BAA since October 2025.

OpenAI marks its Realtime endpoint as zero-data-retention eligible, though with limits worth reading: web search with live internet access is not HIPAA-eligible, and data-residency endpoints carry a 10% price uplift for models released on or after March 5, 2026. Because the vendors sign, compliance almost never forces self-hosting on its own.

GDPR points the same way. Article 28 requires a “processor providing sufficient guarantees” under a relationship “governed by a contract,” which a DPA and data-residency controls satisfy for many EU-data scenarios.

For financial services, the FTC Safeguards Rule requires a security program and oversight that service providers safeguard customer information, and banking regulators treat AI vendors as third-party risk to be managed, not infrastructure to be forbidden.

Private connectivity can substitute for ownership. AWS Bedrock supports private VPC connectivity with no internet gateway, and AWS states Bedrock “never shares your data with model providers.” That satisfies many “data cannot traverse the public internet” clauses without self-hosting a model.

What actually forces self-hosting is a short, specific list:

• An air-gap or no-egress mandate. A vendor container with a license proxy that phones home may fail it.
• A data-residency or placement requirement no vendor region meets.
• A security organization that will not approve any subprocessor, even where a BAA is legally sufficient.
• Government frameworks such as FedRAMP, IL4/5, or CJIS where the vendor lacks authorization at your level. FedRAMP’s own scope page notes that “only a federal agency can determine” whether a use case is in scope.
• A customer contract whose security addendum mandates it.

The theater trap

The most common self-hosting mistake is paying for on-prem GPU inference while the same sensitive data leaves through another door. If the call rides a hosted telephony carrier, the raw audio – the PHI or PII itself – already left your building at the carrier. The PSTN is shared public infrastructure; a phone call is not a private network path.

Self-hosting the models behind a hosted carrier, without the carrier BAA or an owned SIP path, buys nothing on the privacy driver that motivated it. The exposure sits in the audio-touching layers – the telephony carrier and STT – not primarily the LLM.

The minimum-viable private build often owns the media path, STT, and redaction, and keeps the hosted LLM under a BAA where only redacted text reaches it. Keeping the “smart” LLM in-house while raw audio flows through a carrier is the opposite of what the requirement asks.

Map every data class – raw audio, transcripts, tool payloads, recordings, logs – against every component before choosing a tier.

What operating it actually takes

Self-hosting deletes vendor failover and hands you the reliability work. A hosted API gives multi-region redundancy and an SLA. When you self-host, a GPU node dying mid-call is your incident, and you build the health checks, autoscaling, provider fallback, and on-call rotation that replace it. That is a reliability downgrade, accepted in exchange for the privacy or cost upside, and it belongs in the decision explicitly.

A production self-hosted stack needs named owners across distinct functions:

• Voice and application engineering: media pipeline, endpointing, barge-in, orchestration, tool calls, fallback.
• ML inference: model serving, quantization, batching, KV-cache and throughput tuning.
• Platform and SRE: GPU nodes, autoscaling, observability, deploys, on-call, disaster recovery.
• Security and compliance: data flows, contracts, retention, access control, audit evidence.
• Evaluation and QA: call review, synthetic conversation tests, regression suites, release gates.

A vendor-self-hosted deployment can run with 2–3 technical owners plus existing platform support. A full open-weight stack needs 3–5 named owners; they need not all be full-time in steady state, but each function needs real ownership. A 30-person company without GPU and SRE maturity should rarely own the full stack unless voice is strategic product infrastructure.

The recurring costs beyond hardware and salary are easy to miss:

• Peak provisioning and idle. You size for a peak of 2–5x sustained and pay for GPUs 24/7. Low average utilization erases the cost advantage.
• Frozen weights. Hosted frontier models improve continuously for free; your self-hosted weights are frozen until you re-tune and re-validate. That is recurring engineering.
• Evaluation on every change. Every model or serving change needs evaluation against a held-out set or you ship regressions to callers.
• Logging is an egress surface. If call logs or evals flow to a hosted tool, that is another BAA point and another leak. Our observability guide covers the tracing and logging design this requires.

Start reversible

The decision does not have to be all-or-nothing on day one. Design the cascade with swappable STT, LLM, and TTS behind clean interfaces, so each component can move on-prem incrementally without a rebuild.

Start hosted or vendor-self-hosted, prove the product, and migrate the regulated layer later when a hard driver or a validated volume justifies it. That lowers the risk of the whole decision and avoids betting the architecture on an unvalidated sessions-per-GPU number.

Before committing to a GPU or vendor contract, run the one-week benchmark: the exact stack under 10, 50, 100, 200, and 500 synthetic sessions, measuring p95 and p99 turn latency, first-audio and first-token latency, GPU utilization, and error rate, with realistic 8 kHz telephony audio and barge-in, not happy-path turns.

The difference between 100 and 250 sessions per GPU is six figures a year at high concurrency. Guessing is not diligence.

If a hard data-boundary rule, steady high volume, or a strategic model-control need points you toward owning this infrastructure, we run an on-prem feasibility assessment: the data-flow matrix, the tier decision, the fleet-sizing benchmark, and the real cost premium, before you commit budget. That is the work we do at Softcery’s on-premises voice infrastructure practice.