Inside Attention-FFN disaggregation: What Groq LP40 will look like
How AFD works, from ByteDance's MegaScale-Infer paper to NVIDIA's Groq LPUs, and why the LP40 roadmap makes the acquisition hard to explain.
Running an LLM in production used to be straightforward, load the whole model onto a bunch of GPUs and run it. But in recent months, disaggregated LLM inference and heterogeneous computing have picked up serious momentum.
For instance, in March of this year there were 3 important announcements:
AWS + Cerebras: accelerating inference with prefill-decode disaggregation. Prefill on AWS Trainium, decode on Cerebras CS-3.
Gimlet Labs + d-Matrix: a partnership showing 2-5x tokens/sec/user from disaggregated speculative decoding. The target model on NVIDIA GPUs, the draft model on d-Matrix’s Corsair accelerator.
NVIDIA + Groq: at GTC this year, NVIDIA revealed Attention-FFN Disaggregation (AFD). Attention on NVIDIA GPUs, the FFN on the Groq LPU.
In all three, the bandwidth-hungry half of the split (decode, draft, FFN) runs on an SRAM-based chip. Cerebras CS-3, d-Matrix Corsair, and the Groq LPU are all built around on-chip SRAM instead of HBM.
I’ve written extensively about the first two disaggregation techniques, Prefill-Decode and Speculative Decoding. This article is about the third — AFD, and is organized like this:
The recipe: What is Attention-FFN disaggregation, and why would anyone split the decode stage in two?
The research: In 2025, almost a year before GTC, ByteDance published a seminal paper, MegaScale-Infer, demonstrating the benefits of AFD in production. We’ll work through what they found.
The deployment: A closer look at the NVIDIA + Groq solution and whether it actually makes sense for NVIDIA.
The roadmap: LP35 is slated for 2027, and LP40 in 2028. I'll predict what the LP40 will look like, and make the case for why it will leave the $20B Groq deal hard to justify.
Let’s dig in.
The Recipe — What is Attention-FFN disaggregation?
Under the hood, an LLM running inference is just a stack of identical layers, executed one after another. Llama-3 8B has 32 layers. A token enters at the bottom, flows up through all 32, and the model reads its next-token prediction off the top, and every layer has the exact same structure.
Each layer is two sub-blocks bolted together, back to back:
An attention block — the part that looks back, working out how the current token relates to everything before it. It reads the KV cache, the running record of every token seen so far, which grows as the conversation progresses.
A feed-forward block (FFN) — the part that transforms, pushing attention’s output through a few large matrices. The FFN is widely considered to be where the model holds its long-term memory.
These two blocks stress the hardware in completely different ways. In Llama-3 8B, attention holds about 1.3B parameters and the FFN about 5.6B (roughly 80% of the model). Yet attention is the memory-hungry one. Its weights are small, but on every token it re-reads the entire KV cache, and that cache grows with the conversation. A 128K-context chat with one user needs about 16 GB. Attention wants memory capacity, and it scales with context.
The FFN is the opposite: fixed, but enormous. It holds 80% of the weights, and on every token it hauls all of them out of memory to do its matrix multiplications but that cost never changes with context. A 3-word prompt and a 3,000-word prompt drag exactly the same weights across the bus. The FFN wants raw memory bandwidth.
So inside every layer, two workloads with opposite appetites are stapled together. Hold that thought because the next architectural trend makes the imbalance far worse.
Mixture-of-Experts (MoE)
Researchers building LLMs kept hitting the a wall. More parameters make a model smarter, but a bigger dense model is more expensive on every token. MoE breaks that link. It replaces the single dense FFN in a layer with a bank of smaller FFNs, called experts, and puts a small router in front that scores each token and sends it to only the top-k most relevant experts. Every expert adds capacity, but since only subset of FFNs fire, the compute per token stays in check.
In DeepSeek-V3 each MoE layer has 256 routed FFNs, and the router lights up just 8 of the 256 per token. So a 671-billion-parameter model does only about 37 billion parameters of work (~5.5%) on any given token. Enormous capacity parked in memory, a sliver of it active at a time.
Attention is unchanged, but the FFN side has exploded, and almost all of it sits idle for any single token. On one GPU, those dormant experts crowd out the memory attention needs for its KV cache, while the GPU’s compute stalls waiting on weights to load. The imbalance from Llama is now extreme, and the GPU is badly underused.
The AFD split
Attention-FFN disaggregation cuts the layer in two and puts each half on the hardware it wants. Attention runs on one pool, GPUs with plenty of HBM for the growing KV cache. The experts (FFNs) run on a separate pool, chosen for weight bandwidth. Each layer, the attention pool does its work and hands the result across the network to the expert pool, which computes and hands it back for the next layer. Each token ping-pongs between the two.
That’s the recipe.
The Research — ByteDance proves AFD in production
Almost a year before GTC, ByteDance published MegaScale-Infer, which first laid out disaggregating attention from FFN and deployed it in production.
They identified three problems with serving MoE models the traditional way:
Fewer FLOPs don’t mean lower cost. MoE grows a model’s parameters without growing its FLOPs, but that doesn’t translate into cheaper serving in practice.
Sparsity starves the FFN. Each expert is assigned less than a quarter of a batch’s tokens, sometimes an order of magnitude less, which drops FFN utilization and leaves you paying for compute you don’t use.
You can’t just batch your way out. A bigger batch would refill the experts, but latency requirements cap it, the KV cache (scarcer still for large MoE) competes for the memory it needs, and it adds communication overhead.
ByteDance’s answer was MegaScale-Infer. A software layer, communication library, and inference framework, which plays the same role as vLLM or NVIDIA’s TensorRT-LLM. It implements AFD and delivers two key benefits:
Independent scaling: The two halves can use different parallelism strategies. Attention replicated with data parallelism, FFNs spread with expert parallelism. Pooling the token streams from many attention replicas into one expert pool gives each expert a large batch again, pushing the FFN back to compute-bound. That’s the cure for the starved experts, made concrete.
Heterogeneous deployment. With the two halves on separate hardware, you can give each the chip it wants. Memory-cost-effective GPUs for attention, compute-cost-effective GPUs for the experts.
To keep both pools busy, MegaScale-Infer uses ping-pong pipeline parallelism — chopping the batch into micro-batches and shuttling them back and forth so compute overlaps with communication. And in place of NCCL, it uses a custom communication library to make the exchange between pools efficient.
In terms of results, ByteDance showed that disaggregating the layer on a single homogeneous hardware type already delivered up to 1.9× the per-GPU decoding throughput of vLLM and TensorRT-LLM. Splitting it across a heterogeneous H20 + L40S GPU mix, lifted throughput per watt a further ~1.72×. And in production, across nearly 10,000 GPUs, MegaScale-Infer cut the cost of serving the same traffic by 1.5–2.0×.
A couple of other papers on AFD worth checking out: Baidu, Huawei.
The Deployment: NVIDIA + Groq — Rubin does attention, Groq does FFN
Now that we've seen how ByteDance deployed AFD, the NVIDIA + Groq setup is easy to overlay on top of it.