Architecture

Softmax costs O(n squared), so what you can afford depends only on how long the axis is. Kimi K3 drops softmax attention from three sequence layers in every four, and in the same model buys it on the depth axis. We derive the delta rule behind Kimi Delta Attention, work out why its decay carries a floor of exactly -5, unroll a residual connection to see what Attention Residuals replaces, and follow quantile balancing back to the dual it solves.

Part 1 of 2. Every attention variant since 2019 fights the same number: KV cache bytes per token. This post traces the first wave of answers, from MHA through MQA and GQA, to DeepSeek-V2’s Multi-head Latent Attention. We end at the 57× cache reduction that comes from caching a low-rank latent and never materializing K or V at inference.

Most enterprises have bought an AI coding agent and are stuck. The ones generating real productivity gains didn’t win by picking a better model. They built a platform around the agent. This post walks through the five control-plane responsibilities that separate the 11% of AI-native orgs from the 95% reporting zero ROI, grounded in public deployments from Block, Shopify, Atlassian, Airbnb, and others.

An interactive technical breakdown of Claude Code’s architecture — from the query loop and five compaction mechanisms to the permission pipeline and feature flags. Based on source code analysis of ~1,884 TypeScript files.

NVIDIA’s Nemotron-3-Super, IBM’s Granite, and AI21’s Jamba all ship hybrid SSM-Transformer architectures in production. This post builds State Space Models from scratch, starting with a single differential equation, and works up through HiPPO, S4, and the three generations of Mamba to explain why.

Production AI agents face infrastructure problems that framework-level code cannot solve: state loss on crashes, LLM API flakiness, debugging non-deterministic behavior, and coordinating human approvals across hours-long runs. This post walks through Temporal’s durable execution model and why companies like OpenAI chose it for their agent infrastructure.

An interactive visual exploration of OpenClaw — the open-source AI agent that broke GitHub. Explore its three-layer architecture, two key primitives, memory system, and composable system prompt.

A deep dive into vLLM’s tiered model integration — from the Transformers fallback that enables zero-day support to the native integration path that makes it fast.

A deep dive into the architecture, design patterns, and engineering decisions behind production-grade agentic code assist solutions. By dissecting OpenHands, we uncover how to build AI agents that safely execute code, manage complex state, and operate reliably in production.