Long-Context

Softmax costs O(n squared), so what you can afford depends only on how long the axis is. Kimi K3 drops softmax attention from three sequence layers in every four, and in the same model buys it on the depth axis. We derive the delta rule behind Kimi Delta Attention, work out why its decay carries a floor of exactly -5, unroll a residual connection to see what Attention Residuals replaces, and follow quantile balancing back to the dual it solves.

An intuitive, visual guide to Rotary Positional Encoding. Why spinning the query and key vectors beats stamping a position number onto them, why a dot product only ever feels the angle between two vectors, and why that hands you relative position for free. The starting point for understanding how LLMs stretch to long context.