⚡ Inference – Maitreya

KV caching, FlashAttention, and continuous batching - how a trained model actually generates tokens fast.

FlashAttention: The Matrix You Never Have to Write Down

llm-systems

inference

attention

Standard attention builds a T×T score matrix, uses it once, and throws it away. At long sequence lengths that intermediate dwarfs everything else in the layer. The obstacle to skipping it is the softmax — and there is a way around that.

Jul 21, 2026

Deep dive

The KV Cache: Build It, Break It, Measure It

llm-systems

inference

Autoregressive decoding recomputes what it already knows. We build a KV cache from scratch on GPT-2, prove it’s exact, break it three ways on purpose, and measure the price you pay for it.

Jul 13, 2026

Deep dive