finegrained_fp8 models are silently mis-dequantized when loaded on CPU if a quantized weight has partial blocks #50603

Activity

1. YancyLii

   added

   bug

   on Nov 11, 2026
2. its-amann commented on Nov 11, 2026

   

   its-amann

   on Nov 11, 2026

   More actions

   i can take this. plan is to make Fp8Dequantize._dequantize_one use the same ceil grid blocks as _dequantize_fp8_weight, raise when the scale grid doesn't match ceil(shape / weight_block_size), and add a CPU load test with partial blocks
3. davrens101 commented on Nov 12, 2026

   

   davrens101

   on Nov 12, 2026

   More actions

   This is the same bug I was going to report, so I'm adding my reproducer here rather than opening a duplicate issue.

   I verified the mechanism against current main (cloned 2026-11-11):

   • Fp8Dequantize._dequantize_one derives the block size by equal division at src/transformers/integrations/finegrained_fp8.py:1152 (block_m = rows // scale_rows) and raises ValueError when the shape is not divisible by the scale grid (line 1148).
   • The partial-block-aware path _dequantize_fp8_weight (lines 285-303) uses repeat_interleave + [..., :rows, :] and a rem branch — i.e. the correct ceil-grid semantics.

   Reproducer (CPU-only, no downloads; same as the one in the issue body):

   import json, tempfile, warnings
   from pathlib import Path
   import torch
   from safetensors.torch import save_file
   from transformers import LlamaConfig, LlamaForCausalLM
   
   FP8_MAX = 448.0
   def quantize_reference(w):
       rows, cols = w.shape
       gr, gc = (rows + 127) // 128, (cols + 127) // 128
       codes = torch.zeros_like(w); scales = torch.zeros(gr, gc); deq = torch.zeros_like(w)
       for i in range(gr):
           for j in range(gc):
               sl = (slice(i*128, (i+1)*128), slice(j*128, (j+1)*128))
               mult = w[sl].abs().max().clamp(min=1e-12) / FP8_MAX
               codes[sl] = torch.clamp(w[sl]/mult, -FP8_MAX, FP8_MAX)
               scales[i,j] = mult
               deq[sl] = codes[sl].to(torch.float8_e4m3fn).float() * mult
       return codes.to(torch.float8_e4m3fn), scales, deq
   
   def build(root, hidden):
       torch.manual_seed(0)
       cfg = LlamaConfig(hidden_size=hidden, intermediate_size=512, num_hidden_layers=2,
                         num_attention_heads=4, num_key_value_heads=4, vocab_size=128,
                         max_position_embeddings=256, tie_word_embeddings=False,
                         architectures=['LlamaForCausalLM'])
       model = LlamaForCausalLM(cfg).to(torch.bfloat16)
       tensors, ground = {}, {}
       for name, t in model.state_dict().items():
           if name != 'lm_head.weight' and any(s in name for s in
                   ('q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj')):
               codes, scales, deq = quantize_reference(t.float())
               prefix = name[:-len('.weight')]
               tensors[name], tensors[prefix+'.weight_scale_inv'] = codes, scales
               tensors[prefix+'.activation_scale'] = torch.tensor(1.0)
               ground[name] = deq
           else:
               tensors[name], ground[name] = t.detach().clone(), t
       d = root / f'ckpt-h{hidden}'; d.mkdir(parents=True)
       save_file(tensors, str(d/'model.safetensors'))
       cd = cfg.to_dict()
       cd['quantization_config'] = {'quant_method':'fp8','activation_scheme':'dynamic','weight_block_size':[128,128]}
       cd['torch_dtype'] = 'bfloat16'
       (d/'config.json').write_text(json.dumps(cd, indent=2, sort_keys=True))
       return d, ground
   
   def load(ckpt, ground):
       m = LlamaForCausalLM.from_pretrained(str(ckpt), torch_dtype=torch.bfloat16)
       n = 'model.layers.0.mlp.gate_proj.weight'
       return int((m.state_dict()[n].float() - ground[n].to(torch.bfloat16).float()).abs().gt(0).sum())
   
   with tempfile.TemporaryDirectory() as td:
       root = Path(td)
       d, g = build(root, 256)
       print('hidden 256 (divisible): wrong elements =', load(d, g))
       d, g = build(root, 200)
       print('hidden 200 (partial):   wrong elements =', load(d, g))

   Expected on CPU-only: hidden 256 (divisible): wrong elements = 0, hidden 200 (partial): wrong elements = 168915 (all 14 quantized tensors wrong; the partial-aware helper gives max diff 0.0).

   I'm not opening a PR: per CONTRIBUTING.md I can only fix an issue I opened that a maintainer has labeled triaged, so I'll wait for that label.

   Disclosure: this comment was drafted with the assistance of an AI coding assistant; I have not run the reproducer locally.
4. davrens101 commented on Nov 12, 2026

   

   davrens101

   on Nov 12, 2026

   More actions

   Follow-up: I have now run a smaller, op-level reproducer on current main (c527bbf, CPU-only, no downloads). My earlier comment said I had not run the checkpoint reproducer, so this one is verified.

   import torch
   from transformers.integrations.finegrained_fp8 import Fp8Dequantize
   w = torch.ones(130, 128).to(torch.float8_e4m3fn)
   s = torch.tensor([[1.0], [2.0]])  # ceil(130/128)=2 row blocks: rows 0-127 -> 1.0, rows 128-129 -> 2.0
   out = Fp8Dequantize(None)._dequantize_one(w, s, output_dtype=torch.float32)
   print("row 65:", out[65, 0].item(), "(expected 1.0)")
   print("wrong rows:", int((out[:, 0] != torch.tensor([1.0] * 128 + [2.0] * 2)).sum()), "of 130")

   Output:

   row 65: 2.0 (expected 1.0)
   wrong rows: 63 of 130
   

   Here 130 % 2 == 0, so the divisibility check passes and the block size comes out as 65 instead of 128. Rows 65-127 are silently scaled with the wrong block.

   @its-amann has said they will take this, so I won't open a competing PR. If that falls through and a maintainer labels the issue triaged, I'm happy to send the fix.

   Disclosure: drafted with the assistance of an AI coding assistant. The output above is from a run I did.