DeepSeek V4.1 Flash: 1M Context on 4 RTX PRO 6000s

Four PCIe Blackwell cards with no NVLink now serve DeepSeek-V4.1-Flash at the full 1,048,576-token window: a cold 988K-token prompt prefills in 149 s (6,618 tok/s) and decode runs at about 212 tok/s.

Eleven weeks ago the same four cards managed 60 tok/s of decode on llama.cpp and could not hold a long context at all. Getting to a full 1M window and then to this speed took two months of hardware work, one unaccounted memory allocation, an uneven expert split that only became possible when a replacement card arrived dead and we ended up with a mismatched set, and a single environment variable we had forgotten to carry across. This is the whole path — including the parts that failed.

The journey: 630 s to 149 s

Click to enlarge

This did not arrive as one good idea. It took roughly eleven weeks, two of them spent on hardware that had nothing to do with the model, and the last three weeks on getting a 1M-token window to fit and then making it fast. Here is the whole path, including the parts that failed.

August: proving it was possible at all

We started on llama.cpp (2026-08-11) with DeepSeek-V4-Flash and the DSpark draft model, because it was the only stack that ran on sm_120 workstation Blackwell at the time. It worked, and it was slow: decode 60.1 tok/s, context in the tens of thousands.

Two things came out of that month:

• The draft model mattered more than the engine. Adding the DSpark drafter took decode from 60.1 to 102.4 tok/s — a 1.7x — on the same weights.
• The rig itself was broken. A PCIe error storm at 23 errors/minute, traced to a bad board and SlimSAS cabling. A new Tyan S8030GM2NE and a cable swap took it to zero. Then amd_iommu=off iommu=pt plus NCCL peer-to-peer took TP4 prefill from 977 to 2,149 tok/s and decode from 75 to 91.3.

That last one is worth sitting with: one kernel parameter more than doubled prefill before any model work at all.

September: the interconnect was the ceiling

By late September we had four cards serving, and the all-reduce was the bottleneck — not compute.

Date	Change	All-reduce bus bandwidth
2026-09-21	NCCL_P2P_LEVEL SYS → PHB	6.21 → 9.31 GB/s
2026-09-28	Broadcom PEX 88096 switch for three GPUs	9.31 → 19.53 GB/s
2026-10-17	NCCL_P2P_LEVEL=PHB for DS4.1	3.83 → 19.57 GB/s

The third row is the one that hurt. We had set PHB for DeepSeek V4 and forgot to carry it to the DS4.1 launcher. For three weeks, traffic to the GPU outside the switch went through host memory. Fixing it on 2026-10-17 cut cold 498K from 150 s to 75 s and cold 988K from 374 s to 223 s — a bigger single win than anything else in this post.

The lesson: a setting you applied to one launcher is not a setting you applied to the rig. Ours sat there silently costing 40% of prefill.

The 2026-09-29 first DS4.1 serve: 400K window, prefill 2,564 tok/s at 350K, decode 96-108 tok/s.

The memory wall: why 400K and not 1M

This is the part that took the longest, and it was not a tuning problem — it was an unaccounted allocation.

Every attempt at 1M died with the same signature: an OOM mid-request, at a point that moved around, and completely immune to mem-fraction-static. We tuned 0.92 → 0.94 and the failure point did not move at all. That should have told us immediately that the memory was being taken by something the budgeter could not see.

We eventually took an allocator snapshot at the moment of the OOM, and the answer was the V4.1 dense prefill indexer: fp32 [chunk × context] score tensors plus bool candidate masks, about 94% of all prefill memory growth, allocated outside mem_fraction_static's accounting.

Upstream had the same bug and had named it (SGLang issue #35201): "the indexer's context-proportional logits allocation is unaccounted by mem_fraction_static — deterministic OOM on long-context prefill." Someone else had already written down our problem.

The growth was linear and large: 13.7 MiB per 1,000 tokens. At 1M tokens that is 13.7 GB of memory nobody had budgeted for.

Two runtime patches fixed it, with no rebuild:

1. A 256 MiB cap on the indexer copy budget.
2. A row-tiled prefill indexer with int32 candidate block ids — a hand-port of upstream SGLang PR #40217.

That moved the window 400K → 600K → 800K → 1M, and the first cold 988K pass took 629.7 s.

The second memory ceiling: uneven experts

With 1M working, a new limit appeared: the three 84 GiB cards ran out of headroom while the 96 GB card sat half idle.

The numbers, measured after a clean boot:

After a clean boot	PRO 6000 (rank 0)	Each 6000D
Free after the main model loads	21.34 GiB	9.52-9.56 GiB
Free after pools and CUDA graphs	15.86 GiB	4.02-4.12 GiB

11.8 GiB of the PRO 6000 was simply unused. On 2026-10-11 a cold 784K prompt died on the 6000Ds for want of under 0.5 GiB while rank 0 had about 16 GB spare.

The fix was an uneven expert split — the 96 GB card takes 108 of the 384 routed experts, the 84 GiB cards 92 each, instead of the stock 96/96/96/96. One expert position across all 40 layers is 0.70 GiB, so moving four of them frees about 2.7 GiB per 6000D.

Click to enlarge

	Equal split	Uneven 108/92/92/92
6000D main model	72.63 GiB	69.97 GiB
Free after pools	4.64-4.77 GiB	7.31-7.44 GiB
Free at peak, cold 988K	~1.3 GB	~3.8 GB

Before shipping it we validated the routing offline: 45 checks that every expert is loaded by exactly one rank and run in the slot it was loaded into, plus a kernel check showing the uneven split is as close to the single-rank result as the equal split is (0.002337 vs 0.002347), with a deliberately-broken control at 0.933535 to prove the test can tell a wrong routing from a right one.

The dead card that turned into an advantage

The plan was four identical RTX 6000Ds. The fourth arrived dead on arrival.

Replacing it with another 6000D meant waiting, so we bought a full 96 GB RTX PRO 6000 instead — the part that was actually in stock. Three cards 12 GB smaller than the fourth is not a configuration anyone designs on purpose; it is what you end up with when one card fails and you take the replacement you can get.

For most of the build it looked like a liability. Stock SGLang assumes identical ranks, refuses to boot on a mixed set, and the smaller cards were the ones running out of memory.

Then the uneven split turned it around. Because the 96 GB card has more room, it can hold more experts than the others — 108 against 92 each, instead of the stock 96/96/96/96. Moving four expert positions off the small cards frees about 2.7 GiB on each, which is the difference between dying on a cold 784K prompt for want of 0.5 GiB and running comfortably.

That headroom is what turned "fast, but capped at ~400K context" into three concurrent 1M sessions. Without one card able to take a heavier share there would have been nowhere to rebalance into — every card equally short, and no fix available.

So the advantage was not luck. The dead card forced a mixed rig; the uneven split is what converted a constraint into capacity. Had the fourth 6000D worked, we would have had four identical 84 GiB cards, less total VRAM, and no slack to redistribute — the 3-slot configuration would not have existed.

The generalisable point: in a tensor-parallel rig, uneven cards are not automatically a problem — they are only a problem if the sharding is even. Stock SGLang assumes identical ranks and will not start when they are not. Split by capability instead of by count and a mixed rig can beat a uniform one, because the larger card absorbs the imbalance the smaller ones cannot.

If a card dies on you mid-build, that is the move: do not match the set, buy the biggest replacement available, and rebalance the shard.

October: caching, chunking, and the engine move

With memory solved, the remaining work was speed and cache.

• 2026-10-04 — prefill chunk 1,024 → 4,096, and Claude Code system messages kept inline. Cold 988K 629.7 → 373.4 s; prompt-cache reuse per turn from 8,192 of ~35K tokens to 99.9%.
• 2026-10-16 — HiCache file tier, 32 cache tails, 2.1M GPU pool. Restoring a 777K conversation after a restart went from 269.7 s of re-prefill to 5.1 s from disk. 2.7M tokens across six agents at 100% reuse.
• 2026-10-18 — DeepSeek-V4.1-Flash-1017 weights (weights-only; a wash on speed, within noise on quality) and the KV file tier moved to a RAID0 of three NVMe drives. A 339K-token prompt came back from cache in 2.90 s against 55.56 s to compute, and it survived a power cycle.
• 2026-10-21 — the engine move. Cold 988K 223 → 149 s, decode ~108 → ~212 tok/s, free VRAM 1.7 → 4.2 GiB.

The full tuning log

Date	Step	What it bought
2026-10-21	Newer SGLang engine (2026-10-19 SM120 build) + our split, indexer budget and add-ons; chunk 2,048 with expandable_segments	Cold 988K 223 → 149 s; decode ~108 → ~212 tok/s; free VRAM 1.7 → 4.2 GiB
2026-10-18	DeepSeek-V4.1-Flash-1017 weights + Oct 16 encoder fix; KV file tier on NVMe RAID0	Same speed and memory; quality within noise; restored 339K prompt in 2.9 s after a power cycle
2026-10-17	NCCL_P2P_LEVEL=PHB for DS4.1 (it had none)	All-reduce 3.83 → 19.57 GB/s; cold 498K 150 → 75 s; cold 988K 374 → 223 s
2026-10-17	/v1/messages compatibility layer for Claude Code	No more rejected-and-retried requests; PDFs, redacted thinking, over-length prompts handled
2026-10-16	HiCache file tier + 32 cache tails + 2.1M GPU pool	Restore 777K after restart 269.7 → 5.1 s; 2.7M tokens across 6 agents at 100% reuse
2026-10-12	Uneven expert split 108/92/92/92	Free VRAM on the 84 GiB cards 1,322 → 3,798 MiB; decode 108.8 → 110.3 tok/s
2026-10-04	Prefill chunk 1,024 → 4,096; Claude Code system messages kept inline	Cold 988K 629.7 → 373.4 s; cache reuse per turn 8,192 of ~35K → 99.9%
2026-10-03	Indexer scratch budget 256 MiB + row-tiled prefill indexer (port of upstream #40217)	Window 400K → 600K → 800K → 1M; first cold 988K pass in 629.7 s
2026-09-29	DS4.1-Flash first serve: SGLang, TP4 + EP4, Engram tables in host RAM	400K window, prefill 2,564 tok/s at 350K, decode 96-108 tok/s
2026-09-28	Broadcom PEX 88096 switch for three GPUs	All-reduce 9.31 → 19.53 GB/s; V4 prefill +51% at 42K, +20% at 672K
2026-09-21	NCCL P2P level SYS → PHB	All-reduce 6.21 → 9.31 GB/s; V4 prefill +14-24%
2026-09-20	New board (Tyan S8030GM2NE), SlimSAS cable swap	PCIe error storm 23/min → 0
2026-08-25	Kernel amd_iommu=off iommu=pt + NCCL peer-to-peer	TP4 prefill 977 → 2,149 tok/s, decode 75 → 91.3
2026-08-11	DeepSeek V4-Flash on llama.cpp with the DSpark draft model	Decode 60.1 → 102.4 tok/s (1.7x)

What actually mattered, ranked

If you take one thing from eleven weeks:

1. NCCL PHB for this model — 374 s → 223 s on cold 988K. One environment variable, forgotten for three weeks.
2. The indexer patches — took the window from 400K to 1M at all. Nothing else mattered until this was solved.
3. The newer engine — 223 s → 149 s and decode 2x. Real, but it needed the memory work underneath to be usable.
4. The uneven expert split — 1.3 → 3.8 GB of headroom on the cards that were the limit.
5. Chunk 4,096 and HiCache — 630 → 373 s, and restarts from 4.5 minutes to 5 seconds.

The pattern: the biggest wins were configuration and accounting, not clever optimisation. The model was always capable of this. We spent most of the time discovering that memory was being spent where we were not looking.

The rig

One workstation, no NVLink: four Blackwell cards on PCIe 4.0, three of them behind a PCIe switch.

Part	What
GPUs (serving)	1x RTX PRO 6000 Blackwell Workstation (96 GB) + 3x RTX 6000D (84 GiB usable), all PCIe 4.0
PCIe topology	GPUs 0-2 behind a Broadcom PEX 88096 switch; GPU 3 on its own CPU root port
CPU / RAM	AMD EPYC 7F52 (16-core Zen 2) on a Tyan S8030GM2NE, 512 GB DDR4-2933 ECC
Kernel	amd_iommu=off iommu=pt pcie_aspm=off pci=realloc
Cache disk	KV file tier on a RAID0 of 3 NVMe (/mnt/scratch, 1000G cap); weights on a 2 TB NVMe
Serving	SGLang main 2026-10-19 (ormandj SM120 image) + our runtime patches, TP4 + EP4, DSpark speculative decoding, fp8 KV
Model	DeepSeek-V4.1-Flash-1017: 40 layers, hidden 5,120, 384 routed experts (6 active); Engram tables (189 GiB) held in host RAM

Why only three cards are behind the switch

Three of the four are the China-market RTX 6000D (84 GiB usable) rather than the full 96 GB part — 28x 3 GB GDDR7 on a 448-bit bus, 156 SMs, cut down for export rules. The plan was four 6000Ds; the fourth arrived dead and the 96 GB card was bought as its replacement. But it also interacts with the motherboard in a way that is easy to miss, and it is the reason this rig is not four cards behind one switch.

The short version: the Tyan S8030GM2NE does not let us tune the PCIe memory-mapped I/O (MMIO) resource allocation. Without that, four large-VRAM Blackwell cards do not fit in the platform's address space behind a switch, so three go behind the switch and the fourth takes its own CPU root port.

The mechanism, as best we can reconstruct it. This part is inference from the platform's behaviour plus public reports, not from a BIOS dump we took ourselves — treat it as a hypothesis to check on your own board.

Each Blackwell card asks the firmware for a very large BAR1 window to expose its VRAM: on the order of 128 GB per card for this class of part. Four of those is roughly half a terabyte of MMIO address space that the firmware must carve out and align, distributing it across root ports without overlap. Server and workstation BIOSes typically default the Above 4GB MMIO Limit to 40-bit (1 TB) — which sounds generous until the allocator's power-of-two alignment eats most of it. When it cannot place every window, the usual result is a POST hang at a D4 "PCI resource allocation error. Out of resources" code, or a card that simply never enumerates.

Boards that expose Above 4GB MMIO Limit and Re-Size BAR in the BIOS let you raise that ceiling (44-bit / 16 TB is the usual recommendation) and fit all four. Our S8030 does not give us that control, so four behind a switch was not an option.

So the topology is a workaround for a firmware limitation, not a design we chose for speed:

• GPU 3 sits on its own CPU root port, outside the switch.
• Traffic to it cannot be a peer-to-peer copy unless NCCL is told the hop is within the host bridge — which is exactly why NCCL_P2P_LEVEL=PHB was worth 40% of prefill on this rig.
• The three switch-attached cards talk PIX/PXB to each other; the fourth is a longer path.

What we would try next (unverified): a board that lets you raise the above-4G MMIO limit and enable Re-Size BAR should allow all four cards behind the PCIe switch. If it does, every pair becomes PIX/PXB, all-reduce stops crossing a root complex, and P2P bandwidth should improve further. We have not measured this — it is a hypothesis from the topology, not a result. It would probably also let us drop the PHB override, since PHB would no longer be the correct level.

Two practical notes if you are building this:

• Check that BIOS setting before you buy the board. On our hardware it is the single difference between a three-card switch and a four-card one.
• The failure is not always obvious. Some boards POST but silently leave a card unassigned; others hang at D4. Check nvidia-smi -L shows all four and lspci -vv shows every BAR with a real current size matching its supported list before you assume the allocation is correct.

What the engine move bought

Same four GPUs, same 1017 weights, same 2.1M-token KV pool — this comparison predates the third slot, so both columns below are the two-slot configuration. The engine changed and every headline number moved.

Click to enlarge

Metric	Old engine (Oct 17-18)	New engine (Oct 21)
Context window	1,048,576	1,048,576
GPU KV pool	2.1M tokens	2.1M tokens
Cold 498K prefill	74.8 s	54.2 s (9,189 tok/s)
Cold 988K prefill	222.5 s	149.3 s (6,618 tok/s), needles 2/2
Decode, short prompt	108.4 tok/s	212.3 tok/s
Decode at ~1.04M context	103.5 tok/s	184.3 tok/s
Least free VRAM at peak	1,706-1,922 MiB	4,210 MiB
Two ~948K prompts at once	not measured	both answered, 259.6 s
Needle, 5 depths at 878,591 tokens	--	5/5
Tool calls / multi-turn / vision	4/4, 4/4, 11/11	4/4, 4/4, 11/11

The new engine is ghcr.io/ormandj/sglang-deepseek-v41-flash-sm120:v0.4.3: SGLang main of 2026-10-19 plus 32 patches, FlashInfer 0.7.2, built for SM120 (Blackwell workstation) cards. The previous engine was a dev build of about 2026-09-10 plus our own patches.

Cold prefill: 223 s to 149 s

Click to enlarge

Measured cold, nothing cached, needles 2/2 at every level on both engines.

Prompt	498K	598K	698K	798K	898K	988K
Old	74.8	96.9	122.4	152.0	186.8	222.5
New	54.2	69.4	86.5	105.8	127.5	149.3

Short prefill improved by a similar margin:

Prompt tokens	4.1K	32.8K	131K	336K
Old	6,190	8,216	8,233	7,526
New	7,882	11,291	11,441	10,662

Decode: roughly doubled everywhere

Click to enlarge

Context	35	11.6K	47.7K	96K	194K	379K	750K	1,036K
Old	108.4	113.0	99.5	113.8	109.8	107.8	112.8	103.5
New	212.3	201.2	216.9	211.5	197.7	198.7	180.6	184.3

The gain holds at 1M context: 184 tok/s where the old engine managed 104. Free VRAM at peak went from 1.7 GiB to 4.2 GiB, which is what makes the deeper KV pool safe.

It did not work out of the box

Four rounds to get there.

1. Stock profile. Prefill was 36-48% faster and decode about 1.9x, but the profile assumes four 96 GB cards. On our 84 GiB 6000Ds it fit only about 400k tokens of KV — no 1M window.
2. Our expert split, ported. The 108/92/92/92 split and the 256 MiB indexer budget, at chunk 2048 with mem 0.915. 1M passed: 988K cold in 149.1 s.
3. Production's chunk 4096. It boots, but long prefill collapses: 500K cold took 278.7 s against 54 s. Not usable on this engine.
4. Production settings with the full pool. Chunk 2048, mem 0.93, the 2.1M cap, all client add-ons, 32 cache tails and the HiCache file tier. Everything passed. This was the two-slot serve that followed; the third slot came afterwards (below), taking the pool to 3.15M at mem 0.942.

Why chunk 4,096 fails here

This is the most interesting part, and it is not a simple "buffer too big".

At 4,096 rows the engine's memory is full from the first chunk. The FlashInfer MoE scratch space is allocated per call with no preallocated buffer; the mHC residual buffers double; and at 4,096+ rows the engine switches to separate large-batch code paths, including a second CUDA stream, which hold memory longer and fragment it. About 1.85 GiB is set aside for pre-recorded decoder-tail CUDA graphs either way.

That triggers a feedback loop. The V4.1 prefill indexer sizes its score tiles from free GPU memory. With almost nothing free, tiles shrink to a few rows — up to about 1,000 tiny kernel launches per index layer, each re-reading the whole context. The cost therefore grows with context: prefill fell from about 11K to about 1.1K tok/s as context grew. At 2,048 about 4 GB stays free, so tiles stay large.

Field data agrees. Others running this model on RTX PRO 6000s use chunk 2,048 in production; on 8 GPUs, 4,096 was not faster (6,301 vs 6,853 tok/s at 128K). Our own measurement: 4,096 would buy only about 4% of prefill even if it fit.

What stock actually gives you

Before any tuning, the engine's own profile is already much faster — and unusable at 1M on our cards. Worth knowing, because it tells you where the gains come from and where the wall is.

Run at the same chunk 4,096 and the same prompt sizes as the old production serve:

Prompt tokens	Old engine (tok/s)	Stock new engine
4.1K	6,190	8,424
8.2K	7,310	10,356
32.8K	8,216	11,816
131K	8,233	11,995
336K	7,526	11,132

Context	Old decode	Stock decode
35	108.4	214.3
96K	113.8	215.7
379K	107.8	200.4

So out of the box: prefill +36-48%, decode about 1.9x. The engine itself is the win. Everything after this is about making it fit.

Why stock does not hold 1M here

Upstream's profile assumes four 96 GB cards. Our three 6000Ds are 85.6 GB, and without an uneven expert split each holds 74.1 GiB of weights instead of production's 72.6 — while the 96 GB PRO 6000 sat with about 17 GB unused.

Five boot attempts, in order:

Boot	Settings	Outcome
1	as shipped	Refused: TP memory imbalance
2	mem 0.93, chunk 8192	KV 664,832, then OOM in warm-up (980 MiB free)
3	mem 0.91	Refused: weights leave no KV below 0.916
4	mem 0.925, chunk 8192	KV 393,984, then OOM in warm-up (248 MiB free)
5	mem 0.925, chunk 4096	Served — KV 399,872, peak 84.65 of 85.65 GB

Boot 5 works, but at ~400k KV instead of 2.1M: no 1M context, no headroom, and no room for a second agent. The speed is real; the capacity is not.

The fix is the uneven expert split (the 96 GB card takes 108 experts, the 84 GiB cards 92 each), which brings the 6000D weights down to 71.5 GiB and hands the difference to the KV pool. That single change is what turns "fast but 400K" into "fast and 2.1M".

The takeaway for your own build

If you are on four 96 GB cards, stock may simply work. On mixed or 84 GiB cards, expect to rebalance experts before anything else — and do not read a stock-profile benchmark as the number you will get, because it is measuring a memory shape your rig may not be able to hold.

The Oct 18 weights

DeepSeek's 1017 release is a weights-only update — same config, tensor names, shards and tokenizer; the shard hashes differ.

On the old engine, original vs 1017 was a wash: cold 988K 253.5 s vs 257.1 s, decode 108.9 vs 106.1 tok/s. Quality on the saturating sets: HumanEval 157/164 vs 156/164, GSM8K 297/300 vs 296/300. DeepSeek's reported gains are on harder reasoning and agentic benchmarks, which these sets do not reach.

We also ported DeepSeek's Oct 16 encoder fix: reasoning-effort levels low/high/max at 50/75/100, a message carrying only tool results no longer starting a new turn, and ns::name tool namespaces. The new engine already includes these.

Tried and rejected: the reference model's fp8 round trip of the attention output before wo_a cost about 13% of decode (106.1 to 92.7 tok/s) with no measurable quality change. Kept off.

Three 1M sessions at once

The engine was already fast with one conversation. The question that actually matters for agent work is different: can this rig hold three full 1M-token conversations at the same time without falling over?

It can. Three slots now serve in production, and 4 slots was tried and rejected.

The trade: pool versus headroom

A slot that can hold a full 1,048,576-token context is not free. Each one costs about 1.7 GB of KV pool plus about 0.07 GB of decode CUDA graphs.

Click to enlarge

Slots	Pool	mem	Free after start-up (6000D / GPU0)
2 (previous)	2,099,968	0.93	5.97 / 6.55 GB
3 (now)	3,149,824	0.942	4.23 / 4.82 GB
4	4,199,936	0.96	2.50 / 3.09 GB

The pool is simply N × 1,050,000. At mem 0.93 the stack builds 2,599,936 tokens uncapped, and each +0.01 of mem fraction adds about 547k.

Why 4 slots failed. The pool fit, and the functional checks passed — tools, multi-turn 4/4, vision 11/11, OCR 3/3. Then the first ordinary 2k-token request after them died:

cutlass_fused_moe: "Tried to allocate 124.00 MiB ... 40.12 MiB is free" on GPUs 1 and 2

The scheduler exited, and every later request was refused. Least free over that phase was 898 MiB on the 6000Ds. 4 slots is not a config that survives real traffic on 84 GiB cards — the pool fits, the workload does not.

What 3 slots actually delivers

Functional: tools pass, multi-turn 4/4, vision 11/11, OCR 3/3, and the expected Anthropic-worded 400 on an over-length prompt.

Decode with several streams at once (short prompts, thinking off — not comparable to the 212 tok/s single-stream figure, which uses a different harness):

Click to enlarge

Streams	Per stream	Aggregate
1	189.6 tok/s	190.0
2	154.4 tok/s	295.7
3	137.4 tok/s	392.4

Aggregate throughput doubles going from one stream to three, while per-stream speed falls about 28%. That is the expected shape and the right trade for agent work: three agents making progress beats one agent at full speed.

Prefill is unchanged. Three slots and two slots produce the same prefill curve, within noise:

Prompt	2 slots	3 slots
131K	11,441 tok/s	11,431
498K cold	54.2 s	54.1 s
988K cold	149.3 s	149.4 s

These cold levels were measured with 3 slots active, not carried over from the 2-slot serve — the needle checks ran on this configuration: 2/2 at every level. Single-stream decode also stayed within decode_bench's ±10 noise (212.2 at short context, 177.1 at 1,036K).

Worth being precise about what was and was not probed, because the two are easy to conflate:

Test	Slots	Needle check
Cold prefill, 131K-988K	3	2/2 at every level
Three concurrent 988K conversations	3	not probed — correctness judged by the answers, not by a needle
Two concurrent ~948K prompts (earlier)	2	not probed

So the 3-slot configuration is needle-verified for single-request cold prefill, which is what the window claim rests on. The concurrent runs prove the scheduler holds three long sessions and returns correct answers; they do not additionally prove needle retrieval while all three are resident. That is a gap in the evidence, not a known failure.

Three real 988K conversations at once — 2.96M tokens, 94% of the pool — all answered correctly:

Session	Prompt tokens	First token	Result
1	988,050	150.2 s	PASS
2	988,055	297.0 s	PASS
3	988,050	443.9 s	PASS

The lesson: slots add decode concurrency, not prefill throughput

Look at those first-token times: 150 s, 297 s, 444 s. The prefills ran one after another, about 6,680 tok/s each.

That is because the 2,048-token prefill chunk is shared by every running request. A later long prompt waits for the earlier ones to finish. So three slots give you three agents thinking at once, but they do not make three long prompts prefill any faster in total.

If your workload is three agents each cold-starting a 900K prompt, expect the last one to wait. If it is three agents on long-lived cached conversations — which is what agent work usually looks like — you get the full benefit, because the prefix cache means you are not re-prefilling at all.

The soak that mattered

Benchmarks are not a production test, so the 3-slot config was run against an agentic soak: three Claude-Code-like sessions warmed to 698K / 754K / 780K tokens (2.23M at once), each turn carrying a 3-15K tool output plus a <system-reminder>, with 2 parallel subagents every 4 turns, and compaction at 800K.

56 requests, 0 errors, PASS.

Compactions	3/3 (S3 804,298 in 26.5 s, S2 802,023 in 11.2 s, S1 802,489 in 6.8 s) — each read 100.0% of its previous prompt from cache
Agent turns at 600K-804K context	22. Median 6.0 s, max 18.5 s
Subagent cold starts	8 (54K-67K), max 25.3 s while queued
Cache reuse	worst 96.2% across 42 judged turns; every long-context turn at 99%+
Least free	GPU0 2,585; 6000Ds 1,960-2,042 MiB

The three sub-99% turns are all small post-compaction contexts (5-10K previous) where the last partial page is not cached — not a real regression.

One measurement trap worth repeating: the first dry run scored cache reuse as cached ÷ whole prompt, which understates reuse whenever a turn adds a lot to a small context (a 3K tool output on a 5K subagent scored 51-60% and FAILed). Scoring against the conversation's previous prompt is the honest measure — the server data was identical in both runs. If you are measuring prefix-cache reuse, decide which denominator you mean before you trust the number.

The final settings

Three environment overrides on the existing launcher — no file change beyond the defaults:

Setting	Value
MAX_RUNNING_REQUESTS / MAX_NUM_SEQS	3
MAX_TOTAL_TOKENS (pool floor)	3,150,000
MEMORY_FRACTION	0.942

Host RAM for the HiCache tier grows with the pool (ratio 2); that is fine with 287 GB available.

The honest limit: clients only benefit if they actually send more than two requests at once. If your agent tooling caps subagent fan-out at 2, the third slot mostly sits idle — it is headroom for the moment two sessions and a subagent overlap, not a throughput multiplier you will always see.

Settings that matter

Setting	Value	Why
Prefill chunk	2,048	4,096 collapses long prefill on this engine (above); 2,048 keeps ~4 GB free so indexer tiles stay large
PYTORCH_CUDA_ALLOC_CONF	expandable_segments:True	Required. Without it the engine stranded ~5.7 GiB as reserved-but-unused and failed even 256-token requests
Startup warm-up	none	Upstream's --warmups serving_coverage runs out of memory on 84 GiB cards
--mem-fraction-static	0.942	With --max-total-tokens 3150000; the cap, not the fraction, sets use
NCCL_P2P_LEVEL	PHB	Lets NCCL use direct GPU-to-GPU copies to the GPU outside the switch (all-reduce 3.83 vs 19.57 GB/s)
KV pool	3.15M tokens, fp8	Three 1M conversations stay resident
--swa-prefix-tails	32	With HiCache the host tier keeps evicted tails
HiCache	file backend, write-through	GPU -> host RAM (4.2M tokens) -> NVMe; conversations survive restarts
Expert split	108 / 92 / 92 / 92	The 96 GB card takes more experts than the 84 GB ones
Indexer budget	256 MiB	Upstream tiles the indexer now; we only cap its score budget
Concurrency	--max-running-requests 3	Three slots; per-stream speed falls ~28% but aggregate roughly doubles

Client add-ons carried over: Claude Code /v1/messages compatibility, mid-conversation system messages kept inline for prefix-cache reuse, sampling defaults 0.95 / 1.0, a reasoning-effort "off" alias, and a repetition-loop guard.

How to get this yourself

If you have four Blackwell workstation cards and want this result, the order of work matters more than any single flag. This is the sequence that actually worked.

1. Get the fabric right first

Everything downstream is bounded by PCIe. On these cards there is no NVLink, so all cross-GPU traffic is PCIe and the topology decides your ceiling.

• Check link width under load, not at idle. One of our cards trained at x8 because of a faulty riser. At idle it looked fine. It cost about 15% of prefill (cold 988K 253.5 s vs 223.0 s) and was invisible until we compared nvidia-smi link width against lspci -vv. Re-check after any time the case is opened.
• Know your topology. nvidia-smi topo -m tells you which cards share a switch and which sit on their own root port. Ours: three behind a Broadcom PEX 88096, one direct.
• Set NCCL_P2P_LEVEL=PHB. Without it, traffic to the GPU outside the switch goes through host memory. On our rig that was all-reduce 3.83 vs 19.57 GB/s — a 5x difference that showed up as 374 s vs 223 s on cold 988K.
• Kernel flags matter. We boot with amd_iommu=off iommu=pt pcie_aspm=off pci=realloc. iommu=pt alone moved TP4 prefill from 977 to 2,149 tok/s on our previous board.

2. Size the KV pool before chasing speed

The 1M window is a memory problem first. Three things unlocked it:

• An uneven expert split. The 96 GB card takes more experts than the 84 GiB ones: 108 / 92 / 92 / 92. This raised least-free-VRAM on the 84 GiB cards from 1,322 to 3,798 MiB.
• Cap the indexer scratch budget at 256 MiB (DSV41_INDEXER_BUDGET_MIB). This is what moved the window 400K -> 600K -> 800K -> 1M.
• Use a tiered cache. HiCache with a file backend took the KV tier to NVMe, so a 777K conversation comes back in 5.1 s instead of 269.7 s of re-prefill.

If you are on 84 GiB cards, expect to tune the split rather than copy one: the ratio is a function of your card mix.

3. Pick the chunk size by measuring long prefill, not short

This is the single easiest mistake to make. Chunk 4,096 is not faster on this engine — it boots, passes short tests, and then collapses on long context (500K cold: 278.7 s vs 54 s at chunk 2,048).

Do not accept a chunk size because it fits at 32K. Measure a 500K+ cold prompt. The failure mode is a feedback loop, not an out-of-memory crash: the indexer shrinks its tiles when free memory runs out, which multiplies kernel launches, which makes the problem worse as context grows.

4. Set the allocator, skip the warm-up

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is required on this engine. Without it we saw about 5.7 GiB stranded as reserved-but-unused, and even 256-token requests failed. Note this is the opposite of our old-engine finding, where it caused runaway generation.
• Do not use upstream's --warmups serving_coverage. It runs out of memory on 84 GiB cards.
• Set --mem-fraction-static 0.942 with --max-total-tokens 3150000. The explicit token cap, not the fraction, is what actually sets your pool.

5. Verify before you trust it

Checks that would have caught our mistakes:

Check	What it caught
nvidia-smi link width under load	A card training at x8 — 15% of prefill
Cold 500K+ prefill, not just short prompts	Chunk 4,096 collapsing on long context
Needle test at multiple depths	Silent context degradation
Least free VRAM at peak	Whether the 1M window is actually safe
Two concurrent long prompts	Behaviour under real agent load

6. Expect to keep your patches

The stock profile is tuned for four 96 GB cards. On a mixed or 84 GiB setup you will carry your own expert split, indexer budget and chunk size. Ours are documented in the settings table above — treat them as a starting point to measure from, not a config to paste.

The good news: this engine already contains the indexer tiling and encoder fixes we had ported by hand, so there is less to maintain than before.

Dead ends

Tried	Result
Chunk 4,096 (new engine)	Boots, but long prefill collapses: 500K cold 278.7 s against 54 s
Upstream startup warm-up	Out of memory on 84 GiB cards
fp8 round trip of attention output	-13% decode, no quality gain
Chunk 12,288 / 16,384 (old engine)	Out of memory on the first chunk
Chunk 10,240 (old engine)	Fits, 8.6% faster at 988K, but the KV pool halves
Chunk 8,192 (old engine)	6.8% faster for 1.6 GB of VRAM; rejected
Bigger indexer budget (1,024 MiB)	39 allocation failures, no speed-up
fp4 indexer flag (old engine)	25-29% slower; the new engine ships it on
expandable_segments (old engine)	Harmful: runaway generation. Inverted on the new engine, where it is required
SGLang custom all-reduce, torch symmetric memory, NCCL NVLS	Not available on 4 PCIe-only SM120 cards

Two hardware lessons

One card was training at PCIe x8 because of a faulty riser. It cost about 15% of prefill (cold 988K 253.5 s against 223.0 s at x16) and was invisible until we checked link width under load. After a riser swap: 221.8 s, decode 108.4. Check nvidia-smi / lspci after any time the case is opened.

PHB came with one hard reset of the host (an AMD "data fabric sync flood"). The restart afterwards was clean, and it is not yet clear whether PHB was involved. Worth knowing before you copy this.

What we have not proven

Honest limits on these numbers:

• Quality sets were not re-run on the new engine. Same weights, and the functional and needle checks pass, but HumanEval/GSM8K on this engine is still to do.
• Cache restore after a restart was not re-measured. The earlier "777K back in 5.1 s" result belongs to the old engine.
• The engine's PCIe-IPC all-reduce never engaged on these mixed cards ("every rank must use identical collective arguments"), so all-reduce runs on NCCL. Every number above is without it — there may be more available.
• Production since Oct 21 evening, not yet soaked. All numbers are benchmark traffic.
• Three concurrent 988K conversations were not needle-probed. All three answered correctly, but needle retrieval while three 1M sessions are resident is untested — see the table in the 3-slot section. Single-request cold prefill on 3 slots is needle-verified at 2/2.

Credit

DeepSeek for DeepSeek-V4.1-Flash-1017. The SGLang project. ormandj for the SM120 image (ghcr.io/ormandj/sglang-deepseek-v41-flash-sm120). The original 4x RTX PRO 6000 DS4.1 setup, 0xSero/deepseek-v4.1-flash-4x-rtx-pro-6000 (now under sybil-solutions), was the old engine's base. SGLang PR #43376 and issue #40877 informed the chunk analysis; upstream PR #40217 was the indexer tiling idea.

Method: scripts/coldctx_run.py for cold prompts with needle checks, scripts/bench_openai_ctx.py for short prefill, scripts/decode_bench.py for decode (median of 2-3 runs of 512 tokens; a single run varies by about ±10 tok/s).