Most optimizations fail loudly. You get a shape error, or a NaN, or a stack trace with a line number in it.

Quantization doesn’t. You quantize a model to int8, everything loads, every tensor is finite, the generated text reads fine, and two weeks later someone notices the eval has moved. There is no line to look at. Nothing threw.

That silence is the whole difficulty, and it’s why this post starts by building the thing from scratch rather than reaching for a library. There are exactly four operations, and once you can see all four you can see where the accuracy went.

TL;DR — A quantization grid is shared by every value in a tensor, and its resolution is set by the largest one. A single value 16× larger than the rest costs everything else about four bits of local resolution. How much that hurts depends sharply on bit width: for the distributions measured below, 8-bit range selection is far more forgiving than 4-bit. But even a perfect range assumes the tensor is one distribution, and it isn’t — a per-tensor scale can leave whole rows near 0 dB SQNR, where the error carries as much energy as the signal, while the aggregate error looks like it only got about twice as bad. The fix is more grids, and which grids you’re allowed to have is decided by the matmul: a scale factors out of a dot product only if it’s constant along the reduction dimension. Activation outliers in transformers live along exactly that dimension, which is why LLM.int8() splits it and SmoothQuant reparameterizes it.

Everything runs on CPU in a few seconds.

import math
import torch

torch.set_num_threads(1)
torch.manual_seed(0)
print("torch", torch.__version__)

torch 2.8.0

One grid, shared by everybody

The four operations

Quantization is a round trip. You have real numbers, you want to store integers, and you want the real numbers back afterwards with as little damage as possible.

Pick a scale s — the real-world size of one integer step. Divide by it, round to an integer, clamp into the chosen code range, and multiply back:

\[q = \mathrm{clamp}\!\left(\mathrm{round}(x/s),\, -n,\, n\right), \qquad \hat x = s\,q\]

That’s it. Everything else in this post is a consequence of how you pick s.

def quantize_dequantize(x, bits=8, clip=None):
    """Symmetric round-trip quantization. Returns (x_hat, scale).

    `clip` is the largest magnitude the grid will represent. Anything beyond
    it is clamped, not represented.
    """
    n = 2 ** (bits - 1) - 1                  # 127 for int8, 7 for int4
    limit = x.abs().max() if clip is None else x.new_tensor(float(clip))

    if limit == 0:                           # an all-zero tensor is legal input
        return torch.zeros_like(x), x.new_tensor(1.0)

    scale = limit / n                        # real-world size of one step
    q = torch.round(x / scale)               # nearest integer code...
    q = torch.clamp(q, -n, n)                # ...and never let one escape the range
    return q * scale, scale

Four lines of arithmetic and three decisions worth explaining, because they are the ones that bite in real code.

Never let a code escape the range before you store it. Round-then-clamp is the clearest ordering and the one used above. What you must not do is skip the clamp: x/scale can land at 127.6, which rounds to 128, one past the end of signed int8. In this reference implementation the value stays a float and nothing visibly breaks. In current PyTorch the cast to int8 wraps it to -128 — a large positive weight silently becomes a large negative one. Other backends may saturate instead, which is exactly why the quantizer should clamp explicitly rather than rely on whatever the storage conversion happens to do. It is a nasty bug precisely because it fires on a handful of values in a handful of tensors.

edge = torch.tensor(127.6)
print(f"rounded, not clamped : {torch.round(edge).item():.0f}"
      f"  ->  cast to int8: {torch.round(edge).to(torch.int8).item()}")
print(f"rounded and clamped  : {torch.clamp(torch.round(edge), -127, 127).item():.0f}")

rounded, not clamped : 128  ->  cast to int8: -128
rounded and clamped  : 127

Handle the zero-range case. A tensor of all zeros gives limit = 0 and scale = 0, and then x / scale is 0/0 = NaN. It happens more than you’d guess — masked heads, unused expert branches, freshly initialized bias vectors.

n = 2^(bits-1) - 1, so int8 uses [-127, 127]. Signed int8 actually holds [-128, 127]. Giving up -128 makes the representable range exactly symmetric about zero; the grid is then centred on zero, so its zero-point is implicitly 0 and no explicit one needs storing. That’s a convention, not a property of the dtype — asymmetric schemes keep the full [-128, 127] and carry an integer zero-point instead. The zero-point is an integer for a reason worth remembering: real zero has to land exactly on a code, or padding and masked positions pick up a small bias that nothing in your test suite is looking for.

Nothing goes wrong

Here is the failure this post is about. A linear layer, its weights quantized to 4 bits and immediately dequantized, run on the same input:

layer = torch.nn.Linear(512, 512, bias=False)
x = torch.randn(64, 512)

with torch.no_grad():
    y_full = layer(x)
    W_hat, _ = quantize_dequantize(layer.weight, bits=4)
    y_quant = x @ W_hat.T

print(f"shapes match      : {y_full.shape == y_quant.shape}")
print(f"all finite        : {torch.isfinite(y_quant).all().item()}")
print(f"max |difference|  : {(y_full - y_quant).abs().max():.4f}")
print(f"relative error    : {(y_full - y_quant).norm() / y_full.norm():.4%}")

shapes match      : True
all finite        : True
max |difference|  : 0.1803
relative error    : 7.0861%

Every check a normal person would run comes back clean. The output is a perfectly ordinary tensor of perfectly ordinary numbers. It is simply a different answer — 7.1% relative error in this run, spread across every element.

So the question this post has to answer is: where did that few percent come from, and what actually controls it?

One outlier, everybody pays

The scale is set by the largest magnitude in the tensor. Every other value inherits the step size that choice implies. So one value can price precision for all the others.

Concretely: take four thousand values clustered around zero, and drop a single large one in.

bulk = torch.randn(4096) * 0.3
bulk_max = bulk.abs().max().item()
n = 127

print(f"the bulk spans +/-{bulk_max:.2f}\n")
print(f"{'tensor max':>12} {'step':>10} {'intervals over the bulk':>26} {'resolution lost':>18}")
baseline = None
for ratio in [None, 4.0, 16.0, 64.0]:                # multiples of the bulk maximum
    tensor_max = bulk_max if ratio is None else bulk_max * ratio
    step = tensor_max / n
    intervals = 2 * bulk_max / step          # how finely the bulk is still divided
    if baseline is None:
        baseline = intervals
    lost = "" if ratio is None else f"{math.log2(baseline / intervals):>15.2f} bits"
    print(f"{tensor_max:>12.2f} {step:>10.5f} {intervals:>26.1f} {lost:>18}")

the bulk spans +/-1.13

  tensor max       step    intervals over the bulk    resolution lost
        1.13    0.00888                      254.0                   
        4.51    0.03553                       63.5            2.00 bits
       18.05    0.14212                       15.9            4.00 bits
       72.20    0.56850                        4.0            6.00 bits

Read the third column carefully, because it is the only quantity here that means anything: the number of quantization intervals that still span the bulk of the data. Without the outlier the bulk gets the full 254 intervals. With one value 16× larger than the bulk maximum, about 16 intervals remain across the same range.

The outlier is placed as a multiple of the bulk maximum, so the arithmetic is exact: the loss is \(\log_2\) of that multiple, and the table shows 2.00, 4.00 and 6.00 bits for ratios of 4, 16 and 64. So the shorthand is worth memorizing:

A value \(k\) times larger than everything else costs everything else \(\log_2 k\) bits of local resolution.

An int8 tensor with one 16× outlier leaves its ordinary values with roughly the local resolution a clean int4 tensor would have given them. You paid for eight bits and, over the range where your data actually lives, you are getting four.

So don’t represent it

If the outlier is what’s expensive, the obvious move is to refuse to represent it — clamp it and accept the error on that one value in exchange for a finer grid for everyone else.

That’s a real trade with two sides. Lowering the clipping threshold shrinks the step, which reduces rounding error for every value inside the range; it also pushes more values outside the range, where the error is no longer bounded by half a step but by however far outside they were. Total error is the sum, so there should be a minimum somewhere, and it should not obviously be at the maximum.

def best_clip(x, bits, n_points=400):
    """Sweep the clipping threshold and report the best fraction of max.

    The lower bound matters: start the sweep at 0.05 and a heavy-tailed
    tensor will report 0.05 as its optimum, which is the edge of the search
    rather than a minimum. 0.005 is low enough that every case below turns.
    """
    max_abs = x.abs().max().item()
    errors = []
    for frac in torch.linspace(0.005, 1.0, n_points).tolist():
        x_hat, _ = quantize_dequantize(x, bits, clip=max_abs * frac)
        errors.append((((x - x_hat) ** 2).mean().item(), frac))
    best_mse, best_frac = min(errors)
    no_clip = ((x - quantize_dequantize(x, bits)[0]) ** 2).mean().item()
    return best_frac, no_clip / best_mse


heavy = torch.distributions.StudentT(2.5).sample((16384,))
spiky = torch.randn(16384) * 0.3
spiky[0] = 30.0

print(f"{'distribution':<22}" + "".join(f"{b:>17}" for b in ["8 bit", "6 bit", "4 bit", "3 bit"]))
for name, tensor in [("gaussian", torch.randn(16384)),
                     ("student-t (heavy)", heavy),
                     ("bulk + one big value", spiky)]:
    row = f"{name:<22}"
    for bits in [8, 6, 4, 3]:
        frac, gain = best_clip(tensor, bits)
        row += f"{frac:>8.2f}x /{gain:>5.2f}x"
    print(row)

distribution                      8 bit            6 bit            4 bit            3 bit
gaussian                  0.96x / 1.06x    0.85x / 1.29x    0.62x / 2.08x    0.50x / 3.07x
student-t (heavy)         0.93x / 1.09x    0.60x / 1.82x    0.21x / 2.72x    0.11x / 2.42x
bulk + one big value      0.92x / 1.08x    0.40x / 1.98x    0.03x / 1.74x    0.02x / 1.61x

The first number in each cell is where the optimum sits as a fraction of the maximum; the second is how much lower the error is there than at the maximum.

At 8 bits, min-max is nearly right. The optimum sits at 0.93–0.96× of the max and buys 6–9%. You could skip the search entirely and lose almost nothing.

At 4 bits, min-max is badly wrong. The gaussian wants 0.62× of its maximum and the error drops 2.1×; the heavy-tailed tensor wants 0.21× — discarding four fifths of its range — for 2.7×. The tensor with a single large value clips hardest of all — 0.03× of its maximum at 4 bits, 0.02× at 3 — which is to say it throws the outlier away almost entirely and quantizes the bulk as though it were never there.

That is the first genuinely useful thing to carry out of this post, and it’s worth stating carefully. This is a statement about how forgiving range selection is at each bit width, for these distributions. It is not a claim that int8 quantization is easy in general — the rest of this post is about a different way int8 breaks that has nothing to do with the range. What it does explain is why aggressive low-bit quantization can make range selection a first-order decision, while for these distributions int8 is far more forgiving.

A tensor is not one distribution

The aggregate lies

Everything so far assumed the values sharing a grid are drawn from one population. Weight matrices are not like that. Different output channels have wildly different scales — attention projections after training, MLP rows feeding different features, anything that’s been through a normalization layer with learned gains.

Build the pathological version and quantize it two ways: one scale for the entire tensor, versus one scale per output row.

W = torch.randn(128, 256) * 0.1
for i, mult in enumerate([0.01, 0.1, 1.0, 10.0]):   # four families of row scale
    W[i::4] *= mult

n4 = 7                                              # int4

per_tensor_scale = W.abs().max() / n4
W_tensor = torch.clamp(torch.round(W / per_tensor_scale), -n4, n4) * per_tensor_scale

per_row_scale = W.abs().amax(dim=1, keepdim=True) / n4
W_row = torch.clamp(torch.round(W / per_row_scale), -n4, n4) * per_row_scale

mse_tensor = ((W - W_tensor) ** 2).mean().item()
mse_row = ((W - W_row) ** 2).mean().item()
print(f"aggregate weight MSE   per-tensor {mse_tensor:.3e}   per-row {mse_row:.3e}"
      f"   ({mse_tensor / mse_row:.1f}x better)")

aggregate weight MSE   per-tensor 8.341e-03   per-row 3.973e-03   (2.1x better)

Two-and-a-bit times better. On that evidence you might reasonably decide per-tensor scaling is a mild inefficiency and move on.

Now look at the same two results per row, in signal-to-quantization-noise ratio — signal power over error power, in decibels, which is the right unit because it is relative to how much signal that row had in the first place:

def sqnr_db(A, A_hat):
    return 10 * torch.log10((A ** 2).sum(1) / (((A - A_hat) ** 2).sum(1) + 1e-30))


print(f"{'row family':>12} {'per-tensor':>14} {'per-row':>12}")
for i, mult in enumerate([0.01, 0.1, 1.0, 10.0]):
    rows = slice(i, None, 4)
    print(f"{'x' + str(mult):>12} {sqnr_db(W, W_tensor)[rows].mean():>11.1f} dB "
          f"{sqnr_db(W, W_row)[rows].mean():>9.1f} dB")

  row family     per-tensor      per-row
       x0.01         0.0 dB      18.0 dB
        x0.1         0.0 dB      18.1 dB
        x1.0         0.1 dB      18.3 dB
       x10.0        16.4 dB      18.2 dB

Three of the four row families come in at or just above 0.0 dB. Zero decibels means the quantization error carries about as much energy as the signal it is supposed to represent — those rows are severely degraded, most of their entries having been rounded to zero or to the same one or two codes. Only the largest family, the one that set the scale, came through — at 16.4 dB.

The per-row scheme gives every family about 18 dB.

The aggregate said “twice as bad.” The per-row view says three of the four row families were severely degraded and one was fine, and the mean-squared error couldn’t see it because the surviving family carries almost all of the squared magnitude. An aggregate metric weighted by signal power is structurally blind to the failure of low-power components. That is worth internalizing well beyond quantization.

So: give the rows their own scales. Which raises the question of what you’re allowed to do.

Which scales factor out, and why

You do not get to put a scale wherever you like, and the constraint is not a convention — it falls out of the matmul.

Fix notation once and use it for the rest of both posts. A linear layer computes

\[Y = XW^\top, \qquad X \in \mathbb{R}^{T \times K}, \qquad W \in \mathbb{R}^{N \times K}\]

T is tokens, K is the input width, N is the output width. The sum runs over K — that’s the reduction dimension, the one that disappears.

Write out a single output element:

\[Y_{tn} = \sum_k X_{tk} W_{nk}\]

Now suppose the weight scale varies by output channel, \(W_{nk} = s_n Q_{nk}\):

\[Y_{tn} = \sum_k X_{tk}\, s_n Q_{nk} = s_n \sum_k X_{tk} Q_{nk}\]

The s_n doesn’t depend on k, so it comes out of the sum. The integer accumulation runs untouched and you multiply once at the end, per output element. Same story for a per-token activation scale \(X_{tk} = a_t P_{tk}\) — a_t doesn’t depend on k either.

But suppose the scale varies along the input channel, \(W_{nk} = s_k Q_{nk}\):

\[Y_{tn} = \sum_k s_k Q_{nk} X_{tk}\]

Now s_k is inside the sum, tangled up with the accumulation. You cannot factor it out. Verify all three numerically rather than taking it on faith:

X = torch.randn(32, 256)
Q = torch.randn(128, 256)

s_out = torch.rand(128) + 0.5          # per output channel  (n)
s_in = torch.rand(256) + 0.5           # per input channel   (k)
a_tok = torch.rand(32) + 0.5           # per token           (t)

# per-output-channel weight scale: factors out after the reduction
lhs = X @ (Q * s_out[:, None]).T
rhs = (X @ Q.T) * s_out[None, :]
print(f"per-output-channel  factors out : {torch.allclose(lhs, rhs, atol=1e-4)}")

# per-token activation scale: also outside the reduction
lhs = (X * a_tok[:, None]) @ Q.T
rhs = (X @ Q.T) * a_tok[:, None]
print(f"per-token           factors out : {torch.allclose(lhs, rhs, atol=1e-4)}")

# per-input-channel weight scale: no way to pull it out
lhs = X @ (Q * s_in[None, :]).T
base = X @ Q.T
c = (base * lhs).sum() / (base * base).sum()      # least-squares best scalar
rhs_attempt = c * base
print(f"per-input-channel   factors out : {torch.allclose(lhs, rhs_attempt, atol=1e-4)}")
print(f"  (even the least-squares scalar c={c:.4f} leaves "
      f"{(lhs - rhs_attempt).abs().max() / lhs.abs().max():.1%} max relative error)")

per-output-channel  factors out : True
per-token           factors out : True
per-input-channel   factors out : False
  (even the least-squares scalar c=0.9734 leaves 35.0% max relative error)

This single fact explains the entire menu of quantization granularities you see in practice:

Per-output-channel weight scales: algebraically separable from the reduction, so a kernel can accumulate in integers and rescale once at the end. That convenience is why they are such a common choice.
Per-token activation scales: also separable, and a common choice for dynamic activation quantization.
Group-wise weight scales along K: not separable. The kernel has to respect group boundaries and apply the corresponding scale during the accumulation; the exact schedule is backend-specific. It’s affordable because the fix-up is amortized over a whole group, but it means group size is a quantization and a kernel decision.
Per-input-channel activation scales: not separable, and this is the one that matters next.

Practical note. When someone says a scheme is “per-channel,” always ask which channel. Per-output-channel on weights separates cleanly from the reduction, which is why it is so widely used. Per-input-channel is a different animal and either needs kernel support or needs to be algebraically moved somewhere else. The word is the same and the cost is not.

The awkward axis

Where transformer activations put their outliers

Weights are the comparatively easy case: they are fixed at inference time so you can spend as long as you like choosing scales, per-output-channel scaling separates cleanly from the reduction, and they are typically easier to calibrate than the activation distributions we are about to look at.

Activations are the hard case, and the reason is specifically where their outliers live. In transformer activations the extreme values are not scattered randomly across the tensor — they concentrate in a small number of hidden feature dimensions, persistently, across tokens and across prompts. Those feature dimensions are the K axis. The awkward one.

Here is what that structure looks like, built synthetically so the mechanism is visible:

T_tok, K_dim = 512, 256
act = torch.randn(T_tok, K_dim)
outlier_dims = [7, 23, 88, 154, 201]
act[:, outlier_dims] *= 14.0                # same dims, every token

per_dim_max = act.abs().amax(dim=0)
order = torch.argsort(per_dim_max, descending=True)

print(f"top 5 dimensions by max magnitude : {order[:5].tolist()}")
print(f"planted outlier dimensions        : {sorted(outlier_dims)}")
print(f"they hold {act[:, outlier_dims].abs().max():.1f} vs "
      f"{act[:, [d for d in range(K_dim) if d not in outlier_dims]].abs().max():.1f} elsewhere")
print(f"fraction of all values above 6.0  : {(act.abs() > 6.0).float().mean():.4%}")

act_int8_global = quantize_dequantize(act, bits=8)[0]     # one scale for the tensor
print(f"\nper-tensor int8 activation SQNR   : "
      f"{sqnr_db(act.T, act_int8_global.T).mean():.1f} dB")

top 5 dimensions by max magnitude : [23, 7, 201, 154, 88]
planted outlier dimensions        : [7, 23, 88, 154, 201]
they hold 61.3 vs 4.4 elsewhere
fraction of all values above 6.0  : 1.2909%

per-tensor int8 activation SQNR   : 17.5 dB

A note on evidence. The tensor above is synthetic and is here to make the shape of the problem visible. The claim that real transformer activations behave this way — that a handful of feature dimensions carry outliers systematically, and that the effect becomes pronounced at multi-billion-parameter scale — comes from the LLM.int8() study, not from this cell. If you run the same measurement on a small local model you may see only a weak version, and that is consistent with what that paper reports about scale; it is not a failed replication. Treat the synthetic tensor as an illustration and the citation as the evidence.

Now put the two facts side by side, because together they are a trap:

The outliers are concentrated in specific K dimensions, so a per-input-channel scale is what would fix them.
A per-input-channel scale is exactly the one that does not factor out of the reduction.

The structure of the problem and the structure of the solution are misaligned. Everything that follows is a way around that.

Route it, or move it

If a scale can’t be applied where the problem is, you have a small number of options. One is to build a kernel that handles it inside the reduction, which is the group-wise route from the last section. The other two are algebraic, and they’re the ones worth seeing side by side: avoid needing the scale, or move it somewhere it is separable.

Route it — split the reduction. The sum over K is linear, so it splits:

\[Y = X_{\text{normal}} W_{\text{normal}}^\top + X_{\text{outlier}} W_{\text{outlier}}^\top\]

Run the first term in int8 and the second in fp16. Since the outlier dimensions are few, the fp16 term is a thin matmul and the overwhelming majority of the arithmetic stays 8-bit. This is LLM.int8()’s mixed-precision decomposition, and once you’ve seen the axis argument it stops looking like a trick.

What the next cell is and isn’t

It isolates the mixed-precision decomposition and nothing else. Production LLM.int8() also uses vector-wise quantization for the ordinary dimensions, so the error reduction below should be read as a demonstration of why separating the outlier dimensions helps, not as a benchmark of the method.

Wq = torch.randn(128, K_dim) * 0.1
mask = torch.zeros(K_dim, dtype=torch.bool)
mask[outlier_dims] = True

reference = act @ Wq.T

# everything in int8, one shared grid
naive = quantize_dequantize(act, 8)[0] @ quantize_dequantize(Wq, 8)[0].T

# split the reduction: ordinary dims int8, outlier dims kept in full precision
a_int8 = quantize_dequantize(act[:, ~mask], 8)[0]
w_int8 = quantize_dequantize(Wq[:, ~mask], 8)[0]
split = a_int8 @ w_int8.T + act[:, mask] @ Wq[:, mask].T

err = lambda y: ((reference - y) ** 2).mean().item()
print(f"all int8, one grid        : {err(naive):.4e}")
print(f"split reduction           : {err(split):.4e}   "
      f"({err(naive) / err(split):.1f}x better)")
print(f"fraction of K kept in fp16: {mask.float().mean():.1%}")

all int8, one grid        : 5.1099e-02
split reduction           : 4.8477e-04   (105.4x better)
fraction of K kept in fp16: 2.0%

Move it — reparameterize the axis. The other option is to notice that the difficulty can be shifted. For any invertible diagonal D over the K dimension:

\[XW^\top = (XD^{-1})(WD)^\top\]

because \((WD)^\top = D W^\top\) and the two diagonals cancel. Over the reals this changes nothing whatsoever — though note the cell below: the two fp32 evaluation paths are not bit-identical, differing by a few times \(10^{-6}\) in absolute terms here. Exact algebra, two different roundings, which is the theme of this whole series arriving in miniature. But the quantizer doesn’t see exact arithmetic — it sees two tensors with different distributions. Choose D to shrink the activation outliers, and the corresponding weight channels grow to compensate. You have moved the hard part from the tensor that’s hard to quantize to the tensor that’s easy.

alpha = 0.5                                 # how much difficulty to migrate
eps = 1e-8
x_max = act.abs().amax(dim=0).clamp_min(eps)   # per input channel
w_max = Wq.abs().amax(dim=0).clamp_min(eps)    # per input channel, on the weights

D = x_max.pow(alpha) / w_max.pow(1 - alpha)    # the SmoothQuant channel scale

X_smooth, W_smooth = act / D, Wq * D

y_before, y_after = act @ Wq.T, X_smooth @ W_smooth.T
print(f"bit-identical in fp32     : {torch.equal(y_before, y_after)}")
print(f"max fp32 difference       : {(y_before - y_after).abs().max():.3e}")
print(f"activation max  {act.abs().max():>7.2f}  ->  {X_smooth.abs().max():>7.2f}")
print(f"weight max      {Wq.abs().max():>7.2f}  ->  {W_smooth.abs().max():>7.2f}\n")

smoothed = (quantize_dequantize(X_smooth, 8)[0]
            @ quantize_dequantize(W_smooth, 8)[0].T)
print(f"all int8, one grid        : {err(naive):.4e}")
print(f"after moving the scale    : {err(smoothed):.4e}   "
      f"({err(naive) / err(smoothed):.1f}x better)")

bit-identical in fp32     : False
max fp32 difference       : 2.861e-06
activation max    61.34  ->     3.86
weight max         0.42  ->     3.86

all int8, one grid        : 5.1099e-02
after moving the scale    : 5.1026e-03   (10.0x better)

This is SmoothQuant’s scaling rule, and the exponent is the dial that decides how the difficulty is divided. The scale is built from both sides — activation range in the numerator, weight range in the denominator — so raising \(\alpha\) migrates more of the burden out of the activations and into the weights, and lowering it does the reverse. Neither extreme is what you want, which is the second recurring theme of this pair of posts —

Equivalent in floating point does not mean equivalent after quantization.

One scoping note, as with the decomposition above: the scaling rule here is SmoothQuant’s, but the quantizers around it are the deliberately primitive one-grid kind from the start of this post, where SmoothQuant is a W8A8 deployment method. The cell shows the mechanism working; it is not a benchmark of the method.

The algebra told us the outlier axis was awkward before either method appeared. LLM.int8() splits that axis. SmoothQuant reparameterizes it. Same fact, two escapes.

What to carry away

The grid is shared and the largest value sets it. A value \(k\times\) larger than the rest costs the rest \(\log_2 k\) bits of local resolution. A 16× outlier leaves roughly int4-like local resolution over the bulk range.
How much a bad range costs depends on the bit width. For the distributions measured here, min-max at 8 bits sits within 6–9% of optimal; at 4 bits, clipping reduces MSE by 1.7–2.7× and the optimum can move far below the observed maximum.
Aggregate error is blind to low-power components. Per-tensor int4 scaling looked about twice as bad in MSE while leaving three of four row families near 0 dB SQNR, where the quantization error carries as much energy as the signal. Report a per-channel relative measure, not just the mean.
The reduction dimension decides which scales are separable. Per-output-channel weights and per-token activations factor out of the sum. Anything varying along K does not, and needs either a kernel or an algebraic detour.
The systematic LLM activation outliers reported by LLM.int8() occur along hidden-feature dimensions — the K axis here, which is exactly the axis whose scales don’t factor out. That misalignment is the whole problem.
Two important algebraic escapes are routing and reparameterization. Split the reduction and pay for a thin high-precision term, or reparameterize with a diagonal that is an identity over the reals and a real change to what the quantizer sees. A kernel that handles the scale inside the reduction is the third option.

If one sentence survives: quantization error is not a property of a tensor, it is a property of a tensor and the grid it was made to share.

Where this goes next

Everything above treats the grid as the thing to get right, and judges it by how well Ŵ reconstructs W. Weight reconstruction is a useful sanity metric, but aggressive low-bit quantization makes its limitation hard to ignore. Four bits provide 16 codewords, and the symmetric convention used here spends one of them to keep the range even, leaving 15 levels for an entire distribution. At that resolution a harder question arrives: the layer does not use W in isolation — it uses XWᵀ. Which weight errors actually matter, and what would you do differently if you optimized for that instead?

That’s the next post, and the answer turns out to depend on structure in the activations that most treatments never mention.

References

Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022 — systematic outlier feature dimensions and the mixed-precision decomposition.
Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, 2023 — migrating quantization difficulty from activations to weights.
Numerical stability in PyTorch — the same subject one layer up, where the arithmetic is exact but the formula isn’t.