Four Mac Studios: A 4x M3 Ultra MLX Cluster

We had four Mac Studios and a problem worth solving.

The problem was this: a 300-billion-parameter model does not fit on one machine. On a single Mac Studio M3 Ultra with 256 GB, you get maybe a 70B at decent precision. To run something genuinely large — the kind of model that changes what you can do — you need more memory than any one computer has.

So we did the thing you do when you have four Macs and no budget for a datacentre. We stacked them on a desk and wired them together.

Six cables and a stupid amount of hope

Four Mac Studio M3 Ultra, 256 GB of unified memory each. A terabyte in total, sitting in a room with no rack, no fans, no air conditioning. Just four very expensive silver boxes and a nest of Thunderbolt cables.

Not a hub. A full mesh — six cables, one directly between every pair of machines. Because if you put them through a switch, everything funnels through one link, and the whole point of four machines is that they talk to each other constantly.

Each link measured 36 to 38 Gbps. The Wi-Fi we used to manage them ran at about 1 Gbps. We had built something roughly forty times faster than the network we were administering it over, and it still would not be fast enough.

The first day was genuinely exciting. We ran the same model on each Mac separately and they matched within 2% — 23.9, 24.1, 24.3, 24.3 tokens per second. Four healthy machines. A blank canvas.

Then we tried to make them work as one, and it fell apart.

Why we built our own

There was already a project doing this. EXO would mesh Macs together and run models across them, and we started there, because starting from someone else's working code is obviously the right move.

The problem was that EXO had gone quiet. Updates stopped landing. And the models we wanted to run — DeepSeek V4, MiniMax-M3 — were not supported, which meant every new model was a wall rather than a download. When the upstream project is not moving and the model landscape is moving fast, you have two choices: wait, or own it.

We owned it.

The build ended up being three layers stacked on each other:

Upstream MLX, plus patches. Not a rewrite — the Apple framework does the real work. We carried a specific MLX build, then applied our own patches on top. Some were small (arrayscache-trimmable, which made the cache trimmable and unlocked prefix caching). Some were not (1111-hybrid-spec-decode, our attempt at speculative decoding, which — spoiler — we ended up throwing away).

mlx-lm, extended for the models we needed. The library shipped no deepseek_v32 chat template and no pipeline-parallel path for it. MiniMax worked only because an minimax_m2 template happened to exist. Getting DeepSeek V4 and MiniMax-M3 to serve meant adding the model families ourselves, then keeping those patches alive against a moving upstream — we had a rebase plan running continuously, because every upstream release threatened to break them.

Our own cluster layer on top. Sharding across nodes, the supervisor, the console, the cache tiers, the OpenAI-compatible API. The part nobody else had.

So "we built our own engine" is accurate but slightly too clean. What we actually did was stand on upstream and rebuild the parts that were not there — and then spend most of the project maintaining the seam between our patches and theirs.

That is also why the engine went with the hardware. It was never a product. It was a set of patches and a cluster manager tuned to exactly these four machines, these models and this moment. Keeping it alive without the hardware would have meant maintaining a fork of a fork for nobody.

What we actually had to patch

The list is longer than we expected, and it explains why "we'll just use EXO" did not survive contact with reality.

The GPU lockup. With RDMA plus MLX_METAL_FAST_SYNCH=1 at four nodes, the GPU would deadlock inside a fence wait. Not slow — dead. It is a known upstream MLX issue (mlx#3142) with no fix in the tree at the time. The only thing that worked was a community fork that waits on the CPU before encoding the GPU fence. We adopted it, validated it, and it held.

The kernel crash. Turn on batching with a head dimension of 256 — which is exactly what Qwen 3.5/3.6 and the 397B use — and the fork's attention kernel crashed as soon as concurrency went above one. We rebuilt the kernel with smaller fp32 tiles and deployed our own wheel to all four nodes. That is a patched Apple kernel running our production traffic.

The RDMA teardown leak. After a serve stopped, the next one would die with Couldn't allocate protection domain, and only a reboot cleared it. The cause: jaccl allocates one RDMA protection domain per rank and MLX exposes no teardown API — it is freed only if the process exits cleanly. We wrote a graceful teardown to force that.

Model support that did not exist. qwen3_moe had no shard() implementation at all, so it could not be split across machines; we wrote one. qwen3_next had the same gap plus a node-count ceiling tied to key-value heads. DeepSeek V4-Flash shipped no chat template, so tool history was silently dropped between turns — we wrote the shim. GLM-5.2's glm_moe_dsa architecture was not recognised at all and needed a patch to route its tool calls correctly.

And a bug we inflicted on ourselves. Our own qwen3_next patch overrode a shared mixture-of-experts class by name. That was correct for one architecture and wrong for another that reuses the same class, because one replicates its shared expert and the other shards it. The result was rank divergence — word salad at four nodes, perfectly fine at one. It took a per-layer logit probe to find, and it taught us a rule we now apply everywhere: never patch a shared model class by name.

None of this was in a release. It was a fork of a fork, plus a dozen local patches, plus a rebase plan that ran continuously because every upstream release threatened to break them.

And EXO still has not fixed any of it

The reason we left is still the reason. Months on, the lockup has no upstream fix, the bd256 kernel crash has no upstream fix, and the models we wanted to run are still not supported. The project did not just stall while we were building — it stayed stalled.

I do not say that as a complaint. Open source is hard, and EXO did something genuinely impressive: it proved that meshing Macs together could work at all, and we started from that. But if you are evaluating this today and wondering whether the upstream path has caught up, our experience says plan on patching.

Credit where it is due

None of this was entirely our own work, and it would be dishonest to imply otherwise. Three fixes came from other people and we should say so.

rltakashige wrote the fork that fixed our GPU lockup — mlx-jaccl-fix-small-recv, commit cc3f3e60. The change is a CPU-side fence wait plus watchdog-headroom sizing for the command buffer. That one commit is the difference between a cluster that dies daily and one that runs for days. We found it, adopted it, validated it against a 129-minute soak, and it held. Without it this project would not have got past week two.

Lazarus-931 fixed a subtler one upstream — mlx-vlm PR #1484, merged 3 July 2026 as a016e09. A single line: async_eval → eval after prefill. Without it, decode could read blank memory before the first token materialised, and the model would collapse into repeating garbage after seventy or eighty tokens — deterministic, reproducible, and maddening to diagnose because the output looked like a model problem rather than a memory-ordering problem. We cherry-picked that fix onto all four nodes.

ml-explore/mlx issue #3142 is where the lockup was first described, by people hitting it in their own projects. It is still open. That thread is why we knew we were not doing something wrong.

And the foundation underneath all of it: Apple's MLX framework, mlx-lm, and mlx-vlm, plus the EXO project, which showed that meshing Macs together was possible at all. We stood on all of it.

The pattern worth noticing: the two fixes that mattered most were one commit and one line, written by people we have never spoken to, in projects we were not paying for. That is what open source actually does for a project like this — it does not give you a finished product, but it gives you the missing piece if you are willing to go and find it.

The week it did not work

Here is what distributed inference actually looks like on day one: you get output that is technically words, but wrong. Looping. Degenerate. Or the cluster simply stops.

Our first working configuration ran at 6.9 tokens per second. Slower than a single Mac. Slower than reading speed. We had quadrupled the hardware and made things five times worse.

Then we switched from TCP to RDMA, and got to 15. Better. Still absurd — a quarter of one machine's speed, using four.

I want to be honest about that week, because every blog post about this kind of project skips it. There were days where the honest summary was "we made it slower again, in a new and interesting way." You start to wonder whether the whole idea is wrong — whether four machines fundamentally cannot behave like one, and everyone who claims otherwise is selling something.

Then we found one environment variable.

One line that changed everything

MLX_METAL_FAST_SYNCH=1.

That is it. That is the whole thing.

What we ran	Decode
TCP ring	6.9 tok/s
RDMA	15 tok/s
RDMA + one env var	50 tok/s

Fifteen to fifty. Three times faster, from a setting we had never heard of, on the same hardware, the same model, the same day.

What it does is remove a Metal synchronisation barrier that runs after every single command. On one Mac, that barrier is invisible — microseconds, thousands of times a second, who cares. Across four Macs, with tens of thousands of small commands per token, it becomes the entire cost of doing business.

I did not believe the number. I assumed we had measured wrong, or the cache was lying to us. So a week later I turned it off again, deliberately, and watched it fall back to 12.4 tokens per second. Then I turned it back on and it went to fifty.

That was the moment the project became real. Not because 50 tokens per second is fast in absolute terms — it is not, by GPU standards — but because it meant the architecture worked. Four computers could be one computer. Everything after that was tuning.

The lesson I would tattoo on anyone building this: find the per-command synchronisation barrier before you tune anything else. We lost a week to everything except the one thing that mattered.

What it could actually do

Once it worked, it worked properly. One user at a time:

Model	Nodes	Decode
Qwen3.6-27B (dense)	1	23.1
Qwen3.6-27B	4	48.6
Qwen3.5-397B (MoE)	4	~41
DeepSeek-V4-Flash	4	38.7
MiniMax-M3 (vision + tools)	4	28.5
GLM-4.7	4	26.3
Qwen3.6-35B-A3B (MoE)	1	83

The 397B number is the one that still makes me smile. Three hundred billion parameters — fourteen times the size of the 27B — running at about 41 tokens per second, only 15% slower than the small model. That is what a mixture-of-experts architecture buys you: 17 billion active parameters doing the work of 397 billion stored.

We were running models on a desk that most people rent datacentre time for.

And then the humbling ones. GLM-4.7 has 92 layers, which means roughly 2,000 GPU kernel launches for every single token. It sat at 21–26 tokens per second no matter what we did. We tried 8-bit, we tried 4-bit — same speed. The model's shape was the ceiling, not its size.

The 35B MoE did 83 tokens per second on one Mac. On four, about 55–66. We had made it worse by adding machines. That is the lesson nobody tells you: a cluster is not a bigger computer. It is a computer that pays a tax on every word it says.

The 233,000-token conversation

The thing we were most proud of was almost invisible: one conversation, growing to 233,000 tokens across eleven turns.

No crashes. No out-of-memory. It just kept going — 22 tokens per second at 27B, 17 at 397B, with the cache hitting 96%.

Context	27B decode	397B decode	Cache hit
19k	41.1	33.7	0%
132k	25.9	18.4	82%
233k	22.3	16.9	96%

That number matters because of what it avoided. A cold 196,000-token prompt took 686 seconds to process. Eleven and a half minutes. Every turn. Nobody does that twice.

Which is the real discovery of this project, and it has nothing to do with speed.

Caching was the whole game

Our coding agents resend the entire conversation on every turn. So what actually determined whether this cluster felt good to use was not tokens per second — it was whether it could avoid re-reading what it had already read.

History	First token, cold	First token, warm
1,000 tokens	1,265 ms	314 ms
4,000 tokens	3,930 ms	322 ms
8,000 tokens	7,024 ms	330 ms

Seven seconds down to a third of a second. 21 times faster, and — this is the part that matters — the warm number stays flat no matter how long the conversation gets. A 100,000-token conversation costs the same to start as a 1,000-token one.

That is the difference between a tool you tolerate and a tool you reach for.

We had to build it ourselves, because the models use an attention variant whose internal state cannot be trimmed, which breaks the built-in cache. Getting it right took longer than we expected, and the first working version restored a 203,000-token prompt from SSD in 1.1 seconds against 1,025 seconds cold.

I remember the first time a restart came back instantly instead of making us wait seventeen minutes. It was a small, disproportionate joy.

And then it kept falling over

Speed was the easy part. Keeping four Macs alive under real work was most of the project.

About once a day, the GPU would lock up. 100% utilisation, stuck inside a wait, cluster dead. No error, no warning — just a machine that stopped. We lived with that for nine days before a community fork fixed it, and after the fix we ran a soak test: 129 minutes, three concurrent clients, 194 of 194 requests, zero lockups. I read that log three times.

A wrong library file cost us days. Tensor-parallel output was garbage, intermittently. We assumed our code. It was one incorrect libjaccl binary. Pinning the right build fixed everything instantly. Days of debugging, one line of config.

There was a silent correctness bug. A patch was splitting a shared expert that should have been copied whole to every machine. The output was plausible. Not wrong-looking — plausible. That is the worst kind of bug, because you only find it by measuring against a reference you trust, and we nearly didn't.

Two power cuts. The three worker machines rebooted both times. A UPS costs less than the afternoon we lost.

Every one of those is in the post because they are the actual work. The exciting parts — the 50 tokens per second, the 397B model running — happened in hours. The stability happened over weeks.

The things that did not work

This is the section I wish someone had written for us.

Speculative decoding made it slower. Not marginally. 0.26 to 0.37 times — a third of the speed. The technique is supposed to help; on this hardware, with this attention variant, the bookkeeping cost more than the tokens it saved. We tried it, measured it, and turned it off.

int8 KV cache cost speed. Saved 45% of memory, lost 4–10% of decode. Apple GPUs have no int8 tensor cores, so the trade is real and the saving is not. Memory we did not need, for speed we did.

Manual compute/communication overlap gave about 5%. MLX already does it lazily. We were adding complexity for noise.

Running the big model on two nodes instead of four pushed each Mac into memory compression and dropped to 18.8 tokens per second. More machines, fewer machines — there is a right answer and it is not always obvious.

Each of these was a day or two of work that produced nothing. I do not regret them. I regret not knowing which ones to skip.

What I would tell you if you asked

Find the synchronisation barrier first. Before the model, before the quantisation, before anything clever. On our cluster it was a single environment variable worth 3×.

Expect prefill to scale and decode not to. Four machines gave us 3× the prompt processing and 2× the generation. Every token has to cross the wire, twice per layer, and that cost does not shrink when you add machines.

Put small models on one Mac. The 35B was faster on one machine than four. A cluster is for the models that genuinely will not fit.

Build the cache before you build the speed. Warm first token at a third of a second is worth more than 20 extra tokens per second, and I did not believe that until I used it.

Buy a UPS. It is the cheapest lesson on this list.

Pin your library versions. One wrong binary, days gone.

How it ended

We built our own console for this — four nodes, a model library, live telemetry. When you run inference on hardware nobody has benchmarked, you need your own instrument, because otherwise every claim you make is a stopwatch and a feeling.

Click to enlarge

It worked. We served real coding agents from it for two months — long conversations, vision models, tool calls — and it was, genuinely, a pleasure when it behaved.

But Apple Silicon has a ceiling, and we found it. The interconnect is extraordinary for four machines and hopeless for sixteen. The memory is enormous and the compute is not, and there is no version of this where a Mac cluster competes with a GPU rack once the models get big enough. We kept hitting walls that were architectural rather than tunable.

So we moved on. The cluster came apart, the engine went with the hardware, and we went and built something on Blackwell instead — which is a different post, with different problems, and honestly a lot more headroom.

I do not think of it as a failure. We answered the question: yes, four Macs can run a 397B model at 41 tokens per second, and yes, you can hold a 233,000-token conversation on a desk. That is worth knowing, and almost nobody knows it.

But the excitement was in the first two weeks. The other seven were a grind — and if you are about to build one of these, that is the part I would want you to be ready for.

How that went on GPU hardware instead →