oQ+ Quantization

[splatstx]

One question, as I have tried on multiple models now, How do I get the oQ+ Quantization working?

2026-04-16 16:45:15,305 - omlx.oq - INFO - [-] - oQ8: layer sensitivity (descending): L53=0.0002, L59=0.0001, L58=0.0001, L41=0.0001, L57=0.0001, L47=0.0000, L52=0.0000, L1=0.0000, L29=0.0000, L40=0.0000, L56=0.0000, L55=0.0000, L35=0.0000, L0=0.0000, L39=0.0000, L54=0.0000, L49=0.0000, L42=0.0000, L51=0.0000, L43=0.0000, L48=0.0000, L28=0.0000, L38=0.0000, L26=0.0000, L44=0.0000, L34=0.0000, L46=0.0000, L30=0.0000, L27=0.0000, L31=0.0000, L37=0.0000, L45=0.0000, L36=0.0000, L32=0.0000, L50=0.0000, L23=0.0000, L33=0.0000, L25=0.0000, L12=0.0000, L24=0.0000, L17=0.0000, L5=0.0000, L8=0.0000, L11=0.0000, L22=0.0000, L15=0.0000, L7=0.0000, L6=0.0000, L20=0.0000, L9=0.0000, L3=0.0000, L19=0.0000, L21=0.0000, L10=0.0000, L4=0.0000, L18=0.0000, L13=0.0000, L2=0.0000, L16=0.0000, L14=0.0000

[splatstx]

I was able to get it working on google\gemma-4-26B-A4B-it

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: gemma-4-26B-A4B-it-oQ8e
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128          1525.8       13.99   671.1 tok/s    72.0 tok/s       3.303   348.8 tok/s    25.73 GB
pp4096/tg128          4740.1       15.16   864.1 tok/s    66.5 tok/s       6.665   633.8 tok/s    26.19 GB
pp8192/tg128          8647.6       17.01   947.3 tok/s    59.3 tok/s      10.808   769.8 tok/s    26.28 GB
pp16384/tg128        17621.3       22.10   929.8 tok/s    45.6 tok/s      20.427   808.3 tok/s    26.69 GB
pp32768/tg128        37512.1       36.02   873.5 tok/s    28.0 tok/s      42.087   781.6 tok/s    27.46 GB
pp65536/tg128        86789.3       60.62   755.1 tok/s    16.6 tok/s      94.488   694.9 tok/s    29.16 GB
pp131072/tg128      237895.2      109.10   551.0 tok/s     9.2 tok/s     251.750   521.2 tok/s    32.72 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          72.0 tok/s     1.00x   671.1 tok/s   671.1 tok/s      1525.8       3.303
2x         102.6 tok/s     1.42x   684.7 tok/s   342.4 tok/s      2840.6       5.487
4x         140.3 tok/s     1.95x   781.5 tok/s   195.4 tok/s      4794.9       8.890
8x         184.9 tok/s     2.57x   775.0 tok/s    96.9 tok/s      9433.9      16.109

[splatstx]

Seems like its been working the past few models. Uploading them here for anyone interested:
https://huggingface.co/splats/models