Waffles is an AI voice agent living on our softcery.com website. She runs on Gemma 4 26B-A4B, hosted on a rented RTX 3090, the card and the flags in lab 038.

Then Daily shipped PhoneLLM Alpha 1, a full-parameter fine-tune of Nemotron 3 Nano 30B-A3B, built for phone agents. On PhoneBench Alpha 1 it scores 72.3, against 72.4 for GPT-5.6 Terra and 58.1 for Gemma 4 31B. That page publishes no judge, no item list and no harness, and its Gemma row states no quant and no thinking setting.

So: does it serve the same card at the same turn budget? This article covers serve and load. Quality goes in a second article, once we have a scorer.

The outcome

On a rented RTX 3090 on vast.ai, $0.26 per hour (~$190/mo + network) + mradermacher/phonellm-alpha-1-i1-GGUF, the IQ4_XS file, llama.cpp b10902:

llama-server \
  -m phonellm-alpha-1.i1-IQ4_XS.gguf \
  -ngl 99 -c 393216 -np 12 -ub 2048 --cache-type-k f16 --cache-type-v f16 -fa on \
  --no-cache-idle-slots \
  --jinja --chat-template-kwargs '{"enable_thinking": false}' \
  --temp 0 --top-k 40 --top-p 1.0 --min-p 0 --repeat-penalty 1.0

1. Layers, slot cost and the quant

PhoneLLM is a Nemotron 3 Nano hybrid: 52 layers, of which 23 are Mamba2 state layers, 23 are Mixture of Experts, and 6 are attention. Only the 6 attention layers keep a KV cache, one entry per token. The 23 Mamba2 layers keep a state each, a running summary of the history. The 23 states cost 48 MiB per slot to hold. A rewind needs a checkpoint, section 3.

The cost of one caller: 6 attention layers with 2 KV heads give 6 KiB per token, 192 MiB for a 32k slot. The 23 Mamba2 states add 48 MiB per slot in f32. A slot is 240 MiB. The A4B slot is 940 MiB at the same length.

The weights are the second limit. The expert rows are 2,688 and 1,856 wide. Neither divides by 256, so every 256-block quant type falls back to a 32-block type on the 46 expert tensors. For 7 of the 9 public files the experts are IQ4_NL or Q4_0 and weigh 16.5 GB. We read the GGUF headers of the 9 public files over HTTP range requests, no download:

The 7 files differ by 0.3 GB. The quantizer's model card does not say so; it is one template for every model. We took IQ4_XS, the top of the 18 GB group, and there is no smaller quant to try on this card.

Side by side on a 24 GB card:

PhoneLLM needs 3.7 GB more for weights and 8.2 GiB less for 12 slots.

2. The 54 ms floor

A warm turn has 3 parts. The server renders the chat template, it does its other work before the response headers, and it runs the prompt eval over the new tokens. POST /apply-template gives the render alone, with no slot and no prompt eval. timings.prompt_ms gives the prompt eval. The time before the headers includes the render. The turn is that time plus the prompt eval.

On the box, one caller, a cached history of call-1, and a prompt that adds 4 new tokens, 3 reps each:

The prompt eval is 54 to 55 ms on all 6 rows. It holds at 4 and 54 messages, with and without the 5 tool schemas, at -ub 512 and 2048, and at 2 checkpoints and the default. The A4B runs a 1-token prompt eval in 10 to 13 ms. The prompt eval has a fixed cost of 54 ms. We have not found its source. The Mamba2 state restore and the checkpoint save are the candidates, neither profiled.

The prompt eval grows with the new tokens, not linearly. On the replay of call-1, 74 warm turns of 26 to 105 new tokens, the prompt eval is 135.6 ms p50 at 60 tokens. On the conversation run it is 130 ms at 24 tokens. So 54 ms at 4 tokens, 130 at 24, 136 at 60. The step between 4 and 24 is unmeasured. The A4B is at 40 ms for 61 tokens.

Render and before-headers favour PhoneLLM. PhoneLLM's template renders slower than the Gemma one, 35 to 43 ms at 54 messages against 1.6 ms at 82, and the 5 tool schemas add ~30 ms of render. But its total before the headers is lower: 73 to 99 ms at 54 messages with tools, where the A4B spent 126 to 139 ms at 42 messages. The A4B's 2.5 ms per history message does not appear on PhoneLLM. At 4 new tokens the two models reach the first token in about the same time, ~150 ms at 40 to 50 messages with tools, by addition on the A4B side.

So the ~100 ms gap in the warm turn, 188 to 220 ms against 91 to 109, is the prompt eval: the 54 ms floor plus the unmeasured step to 24 tokens. The render and the headers add nothing to the gap.

3. Checkpoints

On the 12B, --ctx-checkpoints 0 cut the warm turn from 556 ms to 72, because the server took 2 or 3 snapshots of 320 MiB on each request and a warm turn did not need them. We carried the flag over to PhoneLLM and ran the same header test as in section 2. All 12 rows came back with cache_n 0:

A repeat of an identical 17,878-token prompt gave the same result: cache_n 0, 8,262 ms.

The cause is the recurrent state. A KV cache holds one entry per token, so the server can cut it at any position and continue from there. A Mamba2 state is a fixed-size summary of the whole history up to a position. To continue from an earlier position, the server needs a copy of the state saved at that position. A checkpoint is that copy. With 0 checkpoints there is no copy, and the server starts from token 0 on each turn, including a turn whose prompt matches the cache in full.

With the default checkpoint count, a turn inside the reach, the distance section 4 measures, reuses 17,362 of 17,878 tokens and re-evaluates 540 to 566, in 170 to 180 ms. A turn past the reach restores the checkpoint at 10,367 tokens, the end of the recorded history, and re-evaluates the 7,500-token tail in 1.5 to 1.7 s.

Things we tried on the same prompt, against the default count:

• --ctx-checkpoints 2. The floor read 55 to 57 ms on 6 rows, against 54 to 55 at the default.
• --ctx-checkpoints 32. The reach stayed at 320 to 360 tokens. A turn past it restored the same checkpoint at 10,367 tokens and re-evaluated 7,691 to 8,311 tokens in 1.6 to 2.4 s. The 30 extra checkpoints did not land inside the 7,500-token tail.

So the count does not move the cost. The default stays. A flag that saved ~480 ms per turn on Gemma costs 2 to 8 s per turn on a hybrid.

4. Barge-in

When the caller interrupts, the agent cuts the reply and sends the next turn with the cut reply in the history. To the server that is a prompt whose prefix ends before the end of the cache. On a KV cache the server drops the tail and continues. On this model it restores the nearest checkpoint before the cut and re-evaluates from there, section 3.

We cached a 17,878-token prompt, the 10,367-token recorded call plus a 7,511-token filler tail, then replaced the last N tokens of the tail and read cache_n, prompt_n and prompt_ms on the next turn. One rep per point, 2 at the default:

Inside the reach the server re-evaluates about one ubatch plus the edit. Past it, cache_n drops to 10,367 on each of the 4 past-reach rows in the table: the server restores the checkpoint at the end of the recorded call and re-evaluates the whole tail, 1.6 to 2.6 s.

On the 3 ubatch settings the restore point inside the reach sits one ubatch before the end of the cache, within 4 tokens: 17,362 at -ub 512, 16,850 at 1024, 15,830 at 2048. The reach is shorter than that distance, 320 to 360 where the boundary is 516 back, and we have not explained the gap. The Gemma article has the same gap at ~400 against 512.

On the A4B the reach is set by the 1,024-token sliding window and does not move with the batch. On PhoneLLM the ubatch sets it. A spoken reply is 6 to 107 tokens on our calls, so a barge-in cut stays inside the reach on both models, at any ubatch from 512 up. The larger ubatch gives a deeper edit: at 2048 a change 1,200 tokens back costs 687 ms, where at 512 it costs a 1.6 s re-evaluation of the tail. A trimmed tool result or a rewritten system line inside 1.2k tokens stays under 0.7 s.

-ub 256 is rejected: the reach falls under 200 tokens and a deep edit costs 2.6 s. We run 2048 for the cold join in section 5.

5. The second caller

A new caller arrives with a prompt that shares nothing with the slots: a cold 10,377-token prompt eval. The server runs all active slots in one batch, so the cold prompt goes through in chunks of one ubatch, and each decode step of a warm caller rides with the next chunk. The warm caller hears the agent stop until the cold prompt is through.

Two warm callers with a cached 10,369-token history and one cold caller, all 3 requests fired at once, on the box, 3 reps:

The stall lands in one of 2 places. In 5 of 6 warm rows per setting the caller waits for the first token. In 1 of 6 the first token comes at 121 to 125 ms and the reply then decodes at 6 to 11 t/s, 1 to 2 times the ~4.5 t/s of speech. The Gemma article saw both shapes, with the stall after the first word in 2 of 3.

At the default ubatch the cold prompt eval runs at 1.2 to 1.5k tokens per second. At 2048 it runs at 2.4 to 3.4k. The cold caller waits half as long, and the warm callers with it. The A4B on the same join, at its 2,048-token chunk, stalls 2.2 to 2.8 s. PhoneLLM at -ub 2048 stalls 2.2 to 3.2 s. At the same chunk size the two models match.

Prefill alone, one caller, one sample: 10,394 tokens in 2,892 ms, 3,594 t/s. The A4B does ~4,200. The sample predates the temperature log, section 7, so its card temperature is unknown.

Not run:

• -b below the ubatch, the fix the Gemma article proposes for the stall. On this model the ubatch also sets the barge-in reach, section 4.
• A cold join against 12 warm callers, the config we run. The rows above are against 2.

6. Callers per card

An idle slot costs 240 MiB of VRAM and no decode time. The cost comes with the active callers, because each decode step runs the whole batch. The card slows as it heats, section 7, so a decode rate from one minute does not compare to a rate from the next. Each load point has a one-caller row measured right before it, and the column to read is the share: the decode per caller over that one-caller rate.

On the 12-slot config, all slots warm on one cached 10,369-token prompt, all callers talking at once, replies of 40 to 51 tokens, 2 reps:

VRAM under the 12 callers: 20,734 MiB, 68 MiB over the load figure, 3,842 MiB free. The slowest of the 24 rows at 12 callers decodes 27 t/s, 6 times the speech rate. Together the 12 produce 480 t/s against 201 alone, by multiplication.

A second sweep on a 24-slot load, same method, took the count to 24. It ran on a hotter card, one-caller decode 100 to 138 t/s against 149 to 201 above, min to p50 in both, so only the shares and the first-token times compare:

The first token grows ~48 ms per added caller, 214 ms at 4 to 1,176 at 24, on both runs. The A4B grew ~16 ms per caller, 149 ms at 1 to 290 at 10. That slope sets the count. VRAM does not: 24 slots load with 1,032 MiB free, and at 24 callers the first token is 1.2 s. The A4B at 10 callers gives 48 t/s per caller at 290 ms; PhoneLLM at 12 gives 40 t/s at 602 ms. We run 12.

The first version of the load script warmed one slot before the run. Each other caller paid a cold 10k prompt eval inside its measured turn, and the first-token maxima read 25 to 74 s. Warm every slot before a load run.

7. Card temperature

The one-caller decode rate on this host fell 55 % inside one run. One 24-minute sweep, same server, same config, a one-caller row of 43 to 46 tokens at the start and at the end:

nvidia-smi set no throttle flag at either point. The clocks and the power show the state: the card holds 1,695 MHz under load and draws 166 W against a 350 W limit.

We then logged the card every 3 s through the replay and the conversation runs. It idles at 57 C and reaches 88 C within 33 s of the first request. The replay log holds one dip to 750 MHz at 86 C. The clock under load sits at 1,695 MHz in the median sample of both logs.

So an absolute rate from a rented card is a function of how long the card has been running. The rule we ended with: log the card, measure a one-caller row next to each load point, and publish the share. That is why section 6 carries shares, and why the decode rows in the outcome table carry a temperature.

Two rows predate the rule. The guidellm sweep, section 10, and the one-sample prefill in section 5 hold an unknown temperature, and we report them as such.

The two ends of the sweep give the gap: 218.6 against 97.6 t/s, 2.2x, on the same server. The numbers do not fit a core limit. 1,695 MHz is the rated boost clock of a 3090, 1.70 GHz on the NVIDIA spec page. The clock fell 11 %. The decode fell 55 %. A card at 1,695 MHz drawing 166 W is idle for part of each step. Decode on this model is bound by memory bandwidth, and the GDDR6X on a 3090 runs 20 to 30 C above the GPU sensor and throttles on its own junction temperature, measured at up to 110 C on these cards. nvidia-smi does not report that sensor, an open request on the NVIDIA developer forum. Hot memory fits the rows. It is a hypothesis: we did not read the memory temperature, clocks.mem or the throttle reasons. Whether a different host holds the cold rate is not measured.

8. Renting

The host passes the same filter as in the Gemma article and runs the same image. It is a 1× RTX 3090 in Czechia at $0.260 per hour. The weight pull is 17.97 GB. We did not time the boot on this build.

The image pins llama.cpp b10902, released 2026-09-11. The 3 upstream changes this model needs all merged before the pin: Nemotron 3 Nano support on 2025-12-16, the Mamba2 shape fix on 2026-06-26 and the checkpoint skip on 2026-06-11. MTP merged on 2026-05-16 as well, and the build does not use it. It carries no draft head: on a hybrid each draft step pays a checkpoint restore, so we did not test it.

The same replay gives 201 ms to the first token over loopback and 311 ms from a laptop: 110 ms per turn for the trip to Czechia. The Gemma host in Sweden cost 150 to 230 ms from the same laptop.

Cost per voice minute on the 12-slot config, the hourly price over 12 callers by division: $0.00036 at 100 % occupancy, $0.00066 at 55 %.

We destroyed the host after the runs. Each open row in section 9 needs a new rental.

9. What we did not measure

• Quality. The replays give counts: 0 empty replies in 396 rows, 3 of 3 tool cases as tool_calls, 0 fixture misses. No scorer, no judge, and the suite prompts a Gemma persona. That is the second article.
• Where the 54 ms floor goes. The Mamba2 state restore and the checkpoint save are the candidates, neither profiled.
• The step in prompt eval between 4 and 24 new tokens, 54 to 130 ms. No row between.
• 8 checkpoints. We have 0, 2, 32 and the default.
• -ub above 2048, and its VRAM cost.
• A cold join against 12 warm callers. The rows are against 2.
• -b below the ubatch, the Gemma article's proposed fix for the stall.
• The cause of the hot card. Memory junction temperature, clocks.mem and the throttle reasons were not logged.
• guidellm on this build. The rows at 1, 3 and 10 callers exist, and hold an unknown card temperature.
• The boot time on this build.
• PhoneBench. The benchmark page publishes no harness, item list or judge names.
• The BF16 baseline and the 24.5 GB Q4_K_M. Neither fits 24 GB.
• A phone-transcript imatrix. It needs a card over 24 GB.
• Anything on a second host or a second prompt. Each number here is one 3090 in Czechia and one recorded call of 10,369 to 10,377 tokens, 1 to 3 reps. The A4B rows come from the same call, 10,335 tokens in its tokenizer.

10. What to try next

In run order:

Not on the list:

• A smaller quant. 7 of the 9 public files hold the same 16.5 GB of experts, section 1.
• The sampler. We run Daily's settings.
• An 8-bit KV cache. The cache is 192 MiB per slot and 3.8 GiB is free.