[WIP] Mimi 4-bit quant on transformer and 8-bit on conv

[iseeyuan]

Summary

[PLEASE REMOVE] See CONTRIBUTING.md's Pull Requests for ExecuTorch PR guidelines.

[PLEASE REMOVE] If this PR closes an issue, please add a Fixes #<issue-id> line.

[PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: " label. For a list of available release notes labels, check out CONTRIBUTING.md's Pull Requests.

Test plan

[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9882

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 72 New Failures

As of commit 8b22cad with merge base 90f0843 ():

NEW FAILURES - The following jobs have failed:

• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels
• Lint / lintrunner / linux-job (gh)
  RuntimeError: Command docker exec -t 2f875953679b9c50818ac523fb96005641749608569f045888710351002dec5b /exec failed with exit code 1
• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 9e59cb15063a64d5145622ed651fd2bbc4a7baee88485d48a5412cdaac084ff7 /exec failed with exit code 1
• pull / test-binary-size-linux / linux-job (gh)
  RuntimeError: Command docker exec -t add6ca9603dda943b70923a97e6328e159aa780483176a2167fd8085f02c1f5c /exec failed with exit code 1
• pull / test-binary-size-linux-gcc / linux-job (gh)
  RuntimeError: Command docker exec -t f1c766e782e0bca4a48be1ee726cdb1c773b28477c223cc477d8b256471e804a /exec failed with exit code 1
• pull / test-custom-ops-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d0ef8e4a03768e2ad020ef32585fdb0e8f9eadbd2111ff31444acd9252570251 /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 0ec6c6a6389daf90ac9ab1599d97f7200257c0251024228e43e1378ab3bfecdd /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t b7bef7b0c231d46074619840550a1f7530cfa3faee0ee8a4b9e9ec6f85545604 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 6cb4f56b99977fd64b71743215f699a7e37d9804d16506cd5c7459502fc14532 /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t 3970da6f08c11b4587146a107e5e5a0e2ac93fddbab01bebbdd1067a15fb5cd7 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t 5f7a591b21fef924154c554d86ce012dbd75310e3db3b999c0ecb16c0f80af04 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
  RuntimeError: Command docker exec -t 2ccf9b244d9ec3addf903b5805d26a5e9ca8fb19239ff32c488c2ab933782b61 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  RuntimeError: Command docker exec -t 6a58ca4ba4c05faacf7b344acbc482c798d145fb01f039ebd1fff90741cdd794 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
  RuntimeError: Command docker exec -t 13325f8d7834edccde700663d30836d627f2691e042ad554362b6cfd78ea9a7c /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t bd9f034a657c66d6bdd9a9db4217faa11acbf476931c4843555ec9d33d4327a4 /exec failed with exit code 1
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
  RuntimeError: Command docker exec -t 71ecc8784abb9241255e7b5e76f87b2236256f7a611517a9331f21e9f821dab7 /exec failed with exit code 1
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command docker exec -t b19f296d907449c746389b1b7dfc7339071c2ac2575b7ab1ae78c094ccb928e4 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t d249a61e5b9e2c2f4f2a52e2162909c859071dc9701f7ee47d4ccfd83146be85 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 3a9bdbeba2f95bd60f745d818eb0b8ada28bde07eb6ed2269c5078f41165a95e /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t afa4e7bae87122dcacf6fc67d01d506a958ff3d8a0e8c433e84ba4ca16c25fa9 /exec failed with exit code 1
• pull / test-models-linux (add_mul, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t e757f493aaeb6c457c2ae6942f86ec527fd6b628a2282c45b746d662db11f323 /exec failed with exit code 1
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 3c5c99799dede42e921a37736c445e8b9c06845bccc253f6b22efcfd56112b67 /exec failed with exit code 1
• pull / test-models-linux (add, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 2e34acc4aa782cf194fb0bcc96e241a6b584df94201a782fb27dbed6f9d14ecd /exec failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t f5325ce38a04f7e9f82ee39268bee082492c868a7fd9f9ce87d5089dc3b9d3b1 /exec failed with exit code 1
• pull / test-models-linux (emformer_join, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c80dd3ffafa7960ef4920ce66192b50435ce8c5df50ec4a99672687ec30bccf /exec failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c91f136714f4baf61bb3128f9a882b4b689f388dfadd7b12009d94837bf8987 /exec failed with exit code 1
• pull / test-models-linux (emformer_transcribe, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 25a4fbccc37fe3af44fd1d1074d0ba988e21c73005d24e1f2e4bbb5fc46fecf4 /exec failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 674b992358dd90d809ab21a982b5afa12a7541063ff595689cf1c438644c5fd3 /exec failed with exit code 1
• pull / test-models-linux (ic3, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 853e2de21339b6c68556fb964521654b01dd0a9ef8328a1ccb45d9010b5b6586 /exec failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 9a7c22ed02effd46e8883dd3805aa5d0fc48c15867054337311f3395f25333e2 /exec failed with exit code 1
• pull / test-models-linux (ic4, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 636fdcef1e718b247419b3c3875c5acdb5ab0a5094a61ef00dfbe02dd03adb90 /exec failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t c52fb1385c1196768f2cb53b3a7c729659cc670a55d775138b8e4b3767802a20 /exec failed with exit code 1
• pull / test-models-linux (linear, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 7b2daccf8eb76f8a05d07c245656338237dc8e80c7885bd53ea304e3066cbd72 /exec failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 79f97c716a51480a0882632fcf29fdd9d6cf877c13e59a0a54d13de81d514e33 /exec failed with exit code 1
• pull / test-models-linux (llama3_2_vision_encoder, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t cd3b81b542a94a269d9a63076394e183ac895a46deff9dd7b6c12e5d308f9ff9 /exec failed with exit code 1
• pull / test-models-linux (mobilebert, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 933c0e06a824c37b514898c3d0110cad3704daace151b9d3b86a1cf8926f80bb /exec failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 780afd252446542893d56f4c9498e117d591c12c5ff3e06645aa7d6bc288423b /exec failed with exit code 1
• pull / test-models-linux (mv2, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 4a0ef935cbc8f2004878ecd6411f930b4b34ade8997df72176405fc5b19ecbe0 /exec failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 0dfaa95d5967062e0b1f8a2099d94e9e7ca169349b55de81d13332eca882e821 /exec failed with exit code 1
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t cc6e6c5d7b6f8b2546d9ee4a22ae31d77b5134b583f70e58b59de3d910777106 /exec failed with exit code 1
• pull / test-models-linux (resnet18, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t c0a21d9dac6f046e5c78ca9dddfc0e1b38c1e8ad3a0863656a74483183079c19 /exec failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 180df6d8b3e5317219477d18248e21ea35937a872fd9e866e30575aa8c895040 /exec failed with exit code 1
• pull / test-models-linux (resnet50, portable, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t f37849d2bb6680cd373f7544be0352e9f9194bd914f2b58b5a2e454bf7b18903 /exec failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 3eac52ea7965d66be44a9ad678746874c65b00ac9ce1fe4dd674e111142034de /exec failed with exit code 1
• pull / test-models-linux (w2l, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 5100542b6843db1ffb78f22589e430293cdfbb8387bcf8b3535a7817f3aaf1c3 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t 8c58b4c1b244624247231a0e39b05ee3594b3687007e560e0e330def9a3fc252 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ef74210888cd4be3aa1123aa9c127db39e48e756e50f2188dc2369e05831e43a /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
  RuntimeError: Command docker exec -t f82fc0a2d819efd9183e0a32607e737ac59e185f7fa01090365be8348b01286b /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 2839c00a1350cb13ac0682ae4fbdc29284e53c888e48ddd810951fbcd74ee2a7 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 7116e26bacf92cba8c922ad0ce7dcdb6723072504d52a97ccbf03abe112a978f /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t f3b2fba0594a72f66dfe43d6c5591902dcb197f07aa0c9ef078d0150b16a91e4 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t bfed59653c25209778f04346031a08f56b8c460e1791d7dbb11ef39a24afef95 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c27dd6ba8b7f71ca1012a7e9626edf19514c78969935e73b31c991a67602455b /exec failed with exit code 1
• pull / test-models-linux-basic (vit, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh)
  RuntimeError: Command docker exec -t b4af787c79ba942b00483b2651c545e342833e0202902d87e07ddacccce6aa58 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 52bdfde7905640180ac05c154cf87ef28540f295be30864546967c4fbec16df2 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 087dfc64288216d2f3cf31f3f953c9b66a6cbfd36fce76c343fa33b91e79568f /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t 5cc724f4006836c64727e7079846c7654c0ad1db6614c3ad961b8912cfbe7630 /exec failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  RuntimeError: Command docker exec -t ca8256791d3f889d84a41978ee1f6c7fb59857bea0768a6a133acbd4e1e03294 /exec failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 5afd1534ee6063f85f9a0ba21cd0bedc404bd48ea83dd9d4cd9f9a55bb88b866 /exec failed with exit code 127
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t c80efe2ef85a1f7ee96efd707d106f091522afb039c9c7eb3cfb8a1ab3a73fbf /exec failed with exit code 1
• pull / test-pybind-build-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 5f1e6536e6fdb47a3aad2c703b6a26749ceec30da21f11584dd965ad7417c745 /exec failed with exit code 1
• pull / test-quantized-aot-lib-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 47658805f393f1ae10c26e0fc7396fdeec945e6e27afe6a3ac67f18f5b44da47 /exec failed with exit code 1
• pull / test-selective-build-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2305a4ae1dcd0c06a03a949aa766b98c214747b7fea3f7e2b5f5d799c544e552 /exec failed with exit code 1
• pull / test-setup-linux-gcc / linux-job (gh)
  RuntimeError: Command docker exec -t 99771eff07e0c5ee43af74f3a003099580c3dd1728b4bf790bb740b8ba0e9eaf /exec failed with exit code 1
• pull / test-static-llama-qnn-linux / linux-job (gh)
  RuntimeError: Command docker exec -t b7035312f2a6db25bfcaf9e85c0fdd539a83890fca2bff8036ad9ceade87002b /exec failed with exit code 1
• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 11df8057491bb66f54d4b9420d956b5c433d878cf3cb72ff49866b783a486784 /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-arm / linux-job (gh)
  RuntimeError: Command docker exec -t fbc6ffdba719b2e814a672990cafbe31b4ae22b62b5cc84b9ece50973627cdc5 /exec failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command docker exec -t d8ad811e9968cf7c87e26bad001e2c44a4eed5c2fce470dcc54e1565ce9f3d45 /exec failed with exit code 1
• pull / unittest-buck / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command docker exec -t f882ed28546bb597dc418b2c21c087ff874817a944d6700b9b055c9a9078d42e /exec failed with exit code 1
• pull / unittest-editable / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[facebook-github-bot]

@limintang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.