Cloud Girl

Cloud Girl

How Does Mistral Large 4 Serve a 1T MoE Model, Explained Visually

Mistral Large 4 keeps a trillion parameters in memory and wakes about 49 billion of them per token. Here's the router, the GPU memory math, the traffic between GPUs and the tricks that keep it fast.

Cloud Girl's avatar
Cloud Girl
Nov 01, 2026
∙ Paid
One token, 128 experts, a trillion parameters on the shelf.
One token, 128 experts, a trillion parameters on the shelf.

On October 29, Mistral put the weights of Mistral Large 4 on Hugging Face, and the release went straight to the top of Hacker News. The headline number is about 1 trillion parameters. For anyone who wants to run it, a smaller number matters more: each token uses about 49 billion of them. Each of the model's 71 mixture-of-experts layers holds 128 experts, and every token wakes up only 4 of them, plus one shared expert that always runs.

One MoE layer: the router lights 4 of 128 experts, plus the shared one, and blends their answers.
One MoE layer: the router lights 4 of 128 experts, plus the shared one, and blends their answers.

That gap between what sits in memory and what does the work is the whole story of mixture-of-experts (MoE) inference. It's why a trillion-parameter model can answer one request at about 65 tokens a second, and also why you still need eight of the biggest GPUs on the market just to hold it. In this post we follow one token through the machine: the router that picks experts, the memory bill, the traffic between GPUs, the hot-expert problem, the sparse attention behind the 1M-token context, and the draft head that roughly doubles speed.


Before we go on: if you're building a career in AI, here's what I'm running right now.

Life of a Model, a free 4-part series. Follow one AI model through its lifecycle: build (pre-training), teach (post-training), tune (fine-tuning, distillation, quantization) and ship (evals, inference, deployment).

Save a free seat

Agentic AI Bootcamp for builders and leaders. A new 5-week cohort starts soon. Structure matters, and so does building alongside people who shape AI strategy at the companies you may want to work for. Readers get 35% off.

Join the Bootcamp (35% off)

  • Storytelling for FDE, 7 hours. The framework that got me into boardrooms with Fortune 100 CEOs and three promotions in five years at Google, shared for the first time. Get $250 off with this link

Now, back to the router.

User's avatar

Continue reading this post for free, courtesy of Cloud Girl.

Or purchase a paid subscription.
© 2026 Priyanka Vergadia · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.