Skip to content

Commit 7d50213

Browse files
chensuyuechensuyue
committed
deploy: 612206c 
1 parent 23cb1fa commit 7d50213

298 files changed

Lines changed: 310 additions & 303 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎latest/_sources/docs/source/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/llama3/README.md.txt‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ In this example, you can verify the accuracy on HPU/CUDA device with emulation o
66

77
```bash
88
# neural-compressor-pt
9-
pip install neural-compressor-pt==3.7
9+
pip install neural-compressor-pt
1010
# auto-round
11-
pip install auto-round==0.9.3
11+
pip install auto-round
1212
# other requirements
1313
pip install -r requirements.txt
1414
```
@@ -79,6 +79,8 @@ Notes:
7979
Here we provide several recipes for Llama3 models. The relative accuracy loss of quantized model should be less than 1%.
8080

8181
> Note: You can also enable static quantization for KV cache by adding `--static_kv_dtype fp8` argument to `quantize.py`， or `--static_kv_dtype=fp8` argument to `run_quant.sh` and `run_benchmark.sh`.
82+
>
83+
> You can also enable static quantization for attention by adding `--static_attention_dtype fp8` argument to `quantize.py`， or `--static_attention_dtype=fp8` argument to `run_quant.sh` and `run_benchmark.sh`. When enabled, it automatically sets KV cache dtype to fp8 as well.
8284

8385
#### Llama 3.1 8B MXFP8
8486

@@ -210,8 +212,10 @@ CUDA_VISIBLE_DEVICES=0,1 bash run_benchmark.sh --model_path=Llama-3.1-70B-MXFP8
210212

211213
The script automatically:
212214
- Detects available GPUs from `CUDA_VISIBLE_DEVICES` and sets `tensor_parallel_size` accordingly
213-
- Runs default tasks: `piqa,hellaswag,mmlu_llama,gsm8k_llama` with batch size 8
215+
- Runs default tasks: `piqa,hellaswag,mmlu_llama,gsm8k_llama` with batch size 64
214216
- Supports custom task selection and batch size adjustment
217+
- Handles special tasks like `mmlu_llama`, `gsm8k_llama` (with chat template) and `longbench` (with extended context length) automatically
218+
- For longbench dataset evaluation, use the `--tasks=longbench` parameter
215219

216220

217221
### NVFP4

‎latest/autoapi/neural_compressor/common/base_config/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -251,7 +251,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
251251
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
252252
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
253253
provided by <a href="https://readthedocs.org">Read the Docs</a>.
254-
<jinja2.runtime.BlockReference object at 0x7f38ba319a00>
254+
<jinja2.runtime.BlockReference object at 0x7ff68732bce0>
255255
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
256256

257257

‎latest/autoapi/neural_compressor/common/base_tuning/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -284,7 +284,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
284284
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
285285
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
286286
provided by <a href="https://readthedocs.org">Read the Docs</a>.
287-
<jinja2.runtime.BlockReference object at 0x7f38b9c01a00>
287+
<jinja2.runtime.BlockReference object at 0x7ff6808c86e0>
288288
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
289289

290290

‎latest/autoapi/neural_compressor/common/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ <h2>Submodules<a class="headerlink" href="#submodules" title="Link to this headi
118118
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
119119
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
120120
provided by <a href="https://readthedocs.org">Read the Docs</a>.
121-
<jinja2.runtime.BlockReference object at 0x7f38b9c5cfb0>
121+
<jinja2.runtime.BlockReference object at 0x7ff6808d34a0>
122122
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
123123

124124

‎latest/autoapi/neural_compressor/common/tuning_param/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
174174
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
175175
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
176176
provided by <a href="https://readthedocs.org">Read the Docs</a>.
177-
<jinja2.runtime.BlockReference object at 0x7f38b9c5cf20>
177+
<jinja2.runtime.BlockReference object at 0x7ff6808d3590>
178178
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
179179

180180

‎latest/autoapi/neural_compressor/common/utils/constants/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
125125
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
126126
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
127127
provided by <a href="https://readthedocs.org">Read the Docs</a>.
128-
<jinja2.runtime.BlockReference object at 0x7f38b9c5d4f0>
128+
<jinja2.runtime.BlockReference object at 0x7ff68093af30>
129129
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
130130

131131

‎latest/autoapi/neural_compressor/common/utils/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ <h2>Submodules<a class="headerlink" href="#submodules" title="Link to this headi
117117
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
118118
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
119119
provided by <a href="https://readthedocs.org">Read the Docs</a>.
120-
<jinja2.runtime.BlockReference object at 0x7f38b9c5e390>
120+
<jinja2.runtime.BlockReference object at 0x7ff680939e50>
121121
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
122122

123123

‎latest/autoapi/neural_compressor/common/utils/logger/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
135135
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
136136
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
137137
provided by <a href="https://readthedocs.org">Read the Docs</a>.
138-
<jinja2.runtime.BlockReference object at 0x7f38b9c5fe30>
138+
<jinja2.runtime.BlockReference object at 0x7ff68093b9b0>
139139
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
140140

141141

‎latest/autoapi/neural_compressor/common/utils/save_load/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
157157
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
158158
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
159159
provided by <a href="https://readthedocs.org">Read the Docs</a>.
160-
<jinja2.runtime.BlockReference object at 0x7f38b9c5cf20>
160+
<jinja2.runtime.BlockReference object at 0x7ff68093b6b0>
161161
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
162162

163163

‎latest/autoapi/neural_compressor/common/utils/utility/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,7 @@ <h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to
278278
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
279279
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
280280
provided by <a href="https://readthedocs.org">Read the Docs</a>.
281-
<jinja2.runtime.BlockReference object at 0x7f38bb9492e0>
281+
<jinja2.runtime.BlockReference object at 0x7ff6808ca870>
282282
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>
283283

284284

0 commit comments

Comments
 (0)