Waffles is an AI voice agent living on our softcery.com website. It was doing just fine until Cerebras decided to remove Gemma 4 from their public pay-as-you-go inference. Classic.

No other provider that we tried was able to match the latency, so we decided to host our own. Below is the result and what we learned.

The outcome

On a rented RTX 3090 on vast.ai, $0.22 per hour (~$150/mo + network) + Unsloth/Gemma-4-26B-A4B-IT-QAT-GGUF:

llama-server \
  -m  gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
  -md mtp-gemma-4-26B-A4B-it.gguf --spec-type draft-mtp --spec-draft-n-max 1 --spec-draft-p-min 0 \
  -ngl 99 -c 98304 -np 3 --cache-type-k f16 --cache-type-v f16 -fa on \
  --ctx-checkpoints 0 --no-cache-idle-slots \
  --jinja --chat-template-file gemma4.jinja --reasoning-format deepseek \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0 --repeat-penalty 1.0

Why the A4B, and why not 16 GB

"A4B" is a Mixture of Experts: 26B parameters on disk, 4B of them used per token. Each of the 30 layers holds 128 expert blocks and routes every token through 8 of them. The experts are 22.8B of the 25.2B parameters. The file loads like a 26B model. Each decode step reads weights like a 4B model.

The Gemma 4 sizes, as Unsloth QAT UD-Q4_K_XL:

Google's model card, thinking on:

Within a point of the 12B on tool calling, above it on knowledge and reasoning, and the 31B is a step above both.

Measured on the 3090, 2 slots of 32k, --swa-full off, no checkpoints. The 12B decode is from a replay of a recorded call, the A4B decode from a live call.

The A4B decodes faster than the 12B and scores above it on the card. The cost is 14.25 GB of weights, which leaves a 16 GB card no room for the KV cache and the runtime. Hence the rented 24 GB.

The 31B is where we came from. Its weights fit a 24 GB card, its KV cache does not: 41 GiB for 2 callers at 24k with every layer keeping the full history. With --swa-full off it comes to ~22 GiB by the formula, and we have not run it.

Fitting the callers into 24 GB

Gemma 4 has two kinds of attention layers. Global layers see the whole conversation. Sliding-window layers see the last 1,024 tokens only. In the 26B-A4B, 25 of 30 layers are windowed. In the 12B, 40 of 48. In the 31B, 50 of 60.

llama.cpp has a flag for what the KV cache keeps for the windowed layers:

• --swa-full on: the full history, as for a global layer. The server can then rewind the conversation to any point. The cache costs 220 KiB per token on the A4B, 336 on the 12B, 880 on the 31B.
• --swa-full off, the default: the window plus one batch, 1,536 cells. The cache costs 20 KiB per token for the 5 global layers on the A4B, 16 on the 12B, 80 on the 31B, plus ~300 MiB per slot for the window cells.

Slots of 32k: the longest call we have recorded is 24.5k tokens, the p99 is 14.6k.

The budget on a 24 GB card, f16 KV cache:

The sum is the 14.25 GB weights, the 250 MB draft head, the cache, and ~570 MiB of runtime as measured on the 12B. The --swa-full on rows are computed from config.json. The same formula came within 0.5% of the measured 12B cache.

With --swa-full off a slot is 940 MiB at 32k, 618 MiB at 16k, and the card takes 10 or 15 of them. An 11th 32k slot leaves under 300 MiB. VRAM under full load, every slot decoding, is 30 MiB over the load figure, so the load figure is the budget. The slot count is not the VRAM question on this model. It is a decode question, covered under the second caller.

With --swa-full on, a 3-bit quant fits: UD-Q3_K_XL at 12.91 GB leaves 450 MiB. We have no quality measurement of a 3-bit A4B, so we keep the 4-bit QAT file and turn --swa-full off.

With --swa-full off, the server cannot rewind the conversation past the window on its own. So by default it snapshots the window cache as the prompt grows, "context checkpoints", and restores the nearest one on a rollback. Each snapshot is a GPU-to-host copy. On the 12B:

The copy runs at ~1.6 GB/s on this host, a PCIe x8 link. The snapshot cost is set by the host bus, not the GPU.

The flag is --ctx-checkpoints 0. The cost: a rollback deeper than ~400 tokens is a full reload of the conversation.

The KV cache stays in f16. An early run with an 8-bit cache and --swa-full on grew VRAM during use. We did not chase it. With --swa-full off the cache is 2.8 GiB at 3 slots, and f16 costs nothing we need.

The chat template, or why the cache breaks

Every turn, the agent sends the whole conversation. The server renders it to one text through the chat template, tokenizes it, and compares the tokens with the slot, the cache of one caller, 3 per card here. The common prefix is reused, the rest is computed. In the response, timings.prompt_n is the count of tokens it had to compute. On a warm turn that number must equal the tokens of the new message. Anything larger is a cache break that far back.

With the template that ships in the Unsloth GGUF, it broke on 4 of 6 turn shapes we tested:

The agent sends what the model produced. The template rewrites it. From that point on the next prompt no longer matches the cache, and the server recomputes everything after it, every turn.

The patched template changes 3 things: items() in place of dictsort for tool arguments, no trim of the model's text, and the text of a tool-call message stays before the call. 6 of 6 shapes give an exact prefix.

On the 12B, 2 slots of 24k with --swa-full on, the same replay on both:

On the A4B, non-QAT build, a live call of 46 requests: 43 warm turns of 17 to 120 new tokens, prompt time p50 31.5 ms, p95 81.9 ms.

Two breaks came from our own agent, not the template:

• The tools arrived on request 2, not request 1. The server recomputed 2,550 tokens, 627 ms. Declare every tool on the first request.
• A float argument the model writes as 1.50 comes back from the parser as 1.5. That is one turn of recompute at that call. We have not seen it in a call yet.

How we found the breaks: a jinja2 render of the stock and the patched template on the 6 shapes, then LLAMA_SERVER_SLOTS_DEBUG=1 and GET /slots to read the text the slot holds. The render finds the 4 in the table. The slot dump found a fifth: the empty thought block the template puts at the end of every prompt and nowhere in the history.

The silence bug

Gemma 4 thinks in a channel: <|channel>thought\n...<channel|> before the reply. With thinking off, Google's template still ends every prompt with an empty block, <|channel>thought\n<channel|>, so the model starts its reply after it. The history turns get no such block.

The reply in the cache sits after the empty block. In the next prompt the same reply sits without it. The prefix ends where the reply starts, and every warm turn recomputes the last reply: prompt_n p50 61.5 tokens where 29 would do, +14 ms p50 on the A4B.

Rendering the block in the history too gives an exact prefix, 6 of 6. We shipped that. Then replies started to come back empty.

The model answers a turn with <|channel>thought\n<channel|> repeated to the token limit. No text. On a replay of a recorded call, 2 of 24 turns did this, 512 tokens in ~2.7 s. On each of the 2 live A4B calls it happened once, 200 tokens at the agent's limit, ~0.9 s. On the phone that is a second of silence and then nothing, and the agent moves on. We heard it as a slow turn before we read the log.

The log signature: a reply at exactly the token limit, the fastest decode of the call, draft acceptance over 0.95, and the next prompt drops the whole reply.

We took the 6 prompts that had triggered it and sampled each 10 times per template:

Things that did not fix it, same 6 prompts:

• A logit bias against token 100, <|channel>. The loop stops. On one prompt, 6 of 10 samples then called a tool with an empty id, and 1 wrote its thinking as reply text.
• A system prompt line against thought channels. Token 100 stayed in 17 of 30 samples.
• A system prompt line to answer the latest message. 13 full loops in 30.
• No empty block at the prompt end either. The model wrote the block itself in 41 of 60 samples, the parser stripped it, and the prefix broke the same way.

So the template stays as Google wrote it on this point: the block at the end of the prompt, none in the history. The 14 ms per turn is the price. On 3 replays of a recorded call, the template without the history block gave 69 of 72 tool calls as recorded, against 64 of 72 with it, and 0 empty replies in 83 rounds against 2 in 88.

Not measured: the 12B showed the signature once in a call with the history block, and we did not measure its rate. And one of the trigger prompts follows a reply cut by a barge-in, and we have not shown that the cut is what triggers it.

The turn budget

A warm turn on the box, from the request to the first token: 91 to 109 ms in 15 reps. The prompt is a recorded call: 10,335 tokens, of which 7.2k the system prompt and 2,613 the tool declarations, in 22 messages, a 38.8 KB request body.

The prompt eval, the part the server log shows, is a tenth of the turn. The pre-header part is the rest, and it depends on the request, not on the model:

Three costs:

• ~27 ms base for a 7.2k-token system prompt, ~10 ms for 2.0k by subtraction of the tool cost.
• ~36 ms for the tool declarations. Likely the grammar the server builds from the tool schemas on every request. Not profiled.
• ~2.5 ms per message in the history, with or without tools, whatever the message length. The template rendered in jinja2 takes 1.6 ms for the same 82 messages, so it is not the template logic. We have not found where it goes.

A voice call is a long list of short messages. At 82 messages the server spends ~270 ms before it starts on the prompt, on a turn whose prompt eval takes 11 ms. Trimming the history breaks the cache prefix, one full recompute per trim, 2.4 s at 10k tokens. We have not settled that trade.

A remote client pays about 3 round trips on top: a TCP connect, then the 38.8 KB body over a fresh connection still in slow start. From a laptop 60 ms away that was 150 to 230 ms over the local figure. The connection is fresh every turn because llama-server closes it after each streamed reply, with a Keep-Alive: timeout=5, max=100 header on the way out. The cause is one line in tools/server/server-http.cpp: the chunked provider calls sink.done() and then returns false, which the vendored httplib reads as a cancel and closes the socket. A return true after done() keeps it open.

Decode is not the constraint. 180 t/s alone, ~94 t/s with 3 callers talking, against ~4.5 t/s of speech. The table by caller count is under the second caller.

Barge-in

When the caller interrupts, the agent cuts the reply at the word the caller heard last and sends the next turn with the cut reply in the history. To the server that is a prompt whose prefix ends before the end of the cache. It has to rewind.

With --swa-full off and no checkpoints, each windowed layer holds 1,536 cells: the 1,024-token window plus one batch of 512. A rewind works while the cells it needs are still there. We edited one word at a set distance before the end of a 10,335-token cached prompt and read the cost:

The cut sits between 390 and ~430 tokens back, in 11 edit points from ~430 to ~4,100. The source puts the reach at one batch, 512 tokens, less what the server generated since. We measure ~400 and have not explained the gap.

A spoken reply is 6 to 107 tokens on our calls, so a barge-in cut stays inside. What does not: any edit further back. A changed system prompt, a trimmed history, a tool result rewritten after the fact. Each of those is 2.4 s at 10k tokens, and on a voice call 2.4 s is a dropped turn.

The rule for the agent: append only, and cut only inside the last reply.

The second caller

llama-server keeps a prompt cache in host RAM, 8 GiB by default, so a caller whose slot was given away can get their context back without a recompute. On every new request, the server saves each idle slot into that cache, unless the cache already holds it. A 10k-token slot on this model saves as 403 MiB: 20 KiB per token for the global layers plus 200 MiB for the 1,024 window cells; the 512 batch cells are not saved. The copy takes 230 to 260 ms, and it blocks the request that triggered it.

With one caller the idle slot never changes and the save runs once. With two callers taking turns, the idle slot has changed every time:

The flag costs nothing we could measure. The cache never restored a context in 6 cases where a caller's slot had gone to a cold prompt: cache_n 0 and a 2.4 s recompute every time, with the flag and without.

Three more cases, 2 slots, on the box, with the flag, 3 reps each:

The stall: the server runs every active slot in one batch. The new caller's 10k tokens go through in chunks of 2,048, ~450 ms each, and each decode step of the warm caller rides with the next chunk. So the warm caller gets a few tokens per chunk until the cold prompt is through. In 2 reps the stall came after the first word, in 1 rep before it. Either way the caller hears the agent stop for 2 to 3 s.

More slots

We loaded 10 slots of 32k and 15 slots of 16k and had every slot talk at once, on the box, 3 reps each, replies of 3 to 43 tokens. An idle slot costs nothing but its VRAM: one caller alone on 10 slots decodes at 218 t/s. The cost comes with the active callers, because each decode step runs the whole batch:

At 15 callers each one still gets 6 times the speech rate. What does not scale is the cold join. On 10 slots, 9 warm callers and one new caller with a cold 11k prompt: the new caller waits 3.3 to 3.6 s for the first token, 18 of the 27 warm turns wait 2.5 to 3.4 s, and the warm decode drops to 10 t/s p50 while the prompt goes through. The same batch, more callers in it. The -b fix in the last section is what makes the count usable.

We run 3. The third caller no longer waits for a reply to end, and 7.3 GiB stays free.

Renting and booting

The card is a 1× RTX 3090 on vast.ai, the cheapest verified offer in the EU that passes a filter: reliability over 0.98, 100 GB disk, CUDA 12.8 or newer, 2 open ports, traffic at most $0.05/GB each way. The numbers behind the filter and the boot:

The image holds llama-server from the upstream CUDA build, aria2c, sshd and our source tree. No weights, no keys. The weights come from Hugging Face at boot, the API key from the environment, the ssh host keys get created at first start. A Hugging Face token does not change the pull speed.

What we did not measure

• The loop rate on live calls after the template fix. The 0 of 60 is on the 6 trigger prompts.
• The 12B loop rate at all.
• Barge-in cut sizes on real calls. The 6 to 107 tokens is the reply size.
• The cold-caller stall with -b 512.
• The slot count against the pre-header cost. One caller on 10 slots saw 149 ms to the first token where 2 slots saw 91 to 109, on a prompt 900 tokens longer. Not isolated.
• The 31B with --swa-full off on this card.
• A 3-bit A4B against the 4-bit QAT on quality.
• Where the 2.5 ms per history message goes in the server.
• Anything on a second host or a second prompt. Every number here is one 3090 in Sweden and one recorded call of 10,335 tokens, 1 to 16 reps.

What to try next

In run order:

Not on the list: a smaller quant, the sampler, an 8-bit KV cache. The card has 7.3 GiB free at 3 slots.