Moe

A mixture-of-experts model decodes efficiently only when each expert processes many tokens for every read of its weights, and top-k routing divides the batch by the ratio of experts to active experts before any expert sees it. DeepSeek serves V4.1-Flash decode on 64 to 128 GPUs because spreading the experts that widely pools every GPU’s tokens into each expert. We derive the quantity that governs this, place one expert on the GB200 roofline, and predict DeepSeek’s measured EP8 decode curve from its EP64 curve to within 6% with no fitted constants.

Softmax costs O(n squared), so what you can afford depends only on how long the axis is. Kimi K3 drops softmax attention from three sequence layers in every four, and in the same model buys it on the depth axis. We derive the delta rule behind Kimi Delta Attention, work out why its decay carries a floor of exactly -5, unroll a residual connection to see what Attention Residuals replaces, and follow quantile balancing back to the dual it solves.