⚡ Inference – Maitreya

KV caching, FlashAttention, and continuous batching - how a trained model actually generates tokens fast.

MLA: Compressing the KV Cache Without Decompressing It

llm-systems

inference

attention

Storing a compressed latent instead of keys and values is easy. Doing it without paying the saving straight back at read time is the hard part — and RoPE is what makes it hard. This post derives absorption from the associativity of a matmul, shows RoPE destroying it with measured errors of 42–100%, and arrives at decoupled RoPE as the repair.

Aug 10, 2026

Deep dive

Speculative Decoding: Exact Sampling Is Only Half the Problem

llm-systems

inference

sampling

A small model proposes, a big model verifies, and the samples are provably from the big model’s distribution. That is the easy half. This post derives the acceptance rule from scratch, shows how correct sampling mathematics can still be defeated by indexing and state inconsistencies, and ends with the region where the algorithm is exact and slower than not using it.

Aug 10, 2026

Deep dive

The Weight Error Is Not the Error That Matters

quantization

numerics

llm-systems

Two int4 quantizers, one with nearly twice the weight reconstruction error, and it is the better one by a factor of three. This post derives what a layer actually experiences, proves exactly when second-order compensation has something to do and when it has nothing, and shows GPTQ confidently correcting for correlations that a finite calibration set invented.

Aug 9, 2026

Deep dive

One Outlier Sets Everyone’s Precision

quantization

numerics

llm-systems

Quantization fails quietly: the shapes are right, the numbers are finite, the text is fluent, and the answer moved. This post builds a quantizer from four operations and follows one question — what does a shared grid cost the values that have to share it — until LLM.int8() and SmoothQuant arrive as two natural responses to it.

Aug 9, 2026

Deep dive

Continuous Batching: The GPU Was Underfilled, Not Underpowered

llm-systems

inference

serving

A decode step costs almost the same for one sequence as for thirty-two, so serving one user wastes most of the machine. Fixing that isn’t a kernel problem — it’s a scheduling one, and the naive fix makes every request wait for the slowest.

Jul 28, 2026

Deep dive

What Ordinary Autograd Saves: Tiling the Forward Pass Is Only Half the Algorithm

llm-systems

training

attention

autograd

A tiled attention forward never builds the T×T score matrix, yet ordinary autograd can retain quadratic aggregate state for the backward pass — more than the dense implementation it replaces. This post measures that behaviour and shows how recomputation changes the scaling.

Jul 21, 2026

Deep dive

FlashAttention: The Matrix You Never Have to Write Down

llm-systems

inference

attention

Attention builds one number for every pair of words, uses it once, and throws it away. At long context lengths that throwaway table dwarfs the tensors that produced it. This post explains why it gets built, why the softmax seems to require it, and the one identity that turns out to make it unnecessary.

Jul 21, 2026

Deep dive

The KV Cache: Build It, Break It, Measure It

llm-systems

inference

Autoregressive decoding recomputes what it already knows. We build a KV cache from scratch on GPT-2, prove it’s exact, break it three ways on purpose, and measure the price you pay for it.

Jul 13, 2026

Deep dive