[Tests]: Adding dummy causal models for testing in regular CI run #427

abukhoy · 2025-05-29T11:42:38Z

Purpose of this PR:

This update aims to reduce test execution time for causal language model inference. Previously, tests were run using full-scale models with one or two layers, which was inefficient and time-consuming. Refactoring CLI api testing for independent testing and redundant conftest code.

What’s Changed:

Introduced dummy models with significantly smaller configurations by adjusting parameters such as max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size and additional_params.
These lightweight models are used exclusively for testing purposes to ensure faster execution without compromising test coverage.

And CLI testing has two test scripts one is for export, compile, and execute, another is for infer cli api.

Note: This optimization is applied only to causal language models.

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-rishinr · 2025-07-03T11:02:56Z

tests/transformers/models/test_causal_lm_models.py

    "hpcai-tech/grok-1",
 ]

+test_dummy_model_configs = [


Can we move this outside this file? may be we can maintain a CSV file for better readability.

Yes, made a json file for dummy configs.

quic-rishinr · 2025-07-03T11:19:36Z

tests/transformers/models/test_causal_lm_models.py

    "hpcai-tech/grok-1",
 ]

+test_dummy_model_configs = [
+    # model_name, model_type, max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size, additional_params
+    ("TinyLlama/TinyLlama-1.1B-Chat-v1.0", "llama", 128, 1, 2, 64, 256, 32000, {"num_key_value_heads": 1}),


are we following any criteria for selecting these configs?

No criteria, but some of the models has constraint for some of the config params with specific value.

quic-rishinr · 2025-07-03T11:55:48Z

tests/transformers/models/test_causal_lm_models.py

-
+    if model_hf is None:
+        model_hf, _ = load_causal_lm_model(model_config)
+    model_hf_cb = copy.deepcopy(model_hf)


why do we need this?

Because, later for CB testing we don't need to call the load_causal_lm_model again. And it reduces the code for dummy model execution. If you don't want this, please let me know.

why we need to have a copy of model_hf for CB?

quic-rishinr · 2025-07-15T05:21:38Z

tests/cloud/test_export_compile_execute.py

+@pytest.mark.cli
+@pytest.mark.parametrize("config", configs)
+def test_export_compile_execute_qnn_fb(mocker, config):
+    # testing export -> compile -> infer with full_batch_size in QNN enviroment


Typo in "enviroment"

quic-rishinr · 2025-07-15T05:25:24Z

tests/cloud/test_export_compile_execute.py

+@pytest.mark.qnn
+@pytest.mark.cli
+@pytest.mark.parametrize("config", configs)
+def test_export_compile_execute_qnn(mocker, config):


Both test_export_compile_execute_qnn and test_export_compile_execute_qnn_fb is currently having same configs right? Ideally in test_export_compile_execute_qnn we should be providing BS and in test_export_compile_execute_qnn_fb we should be providing FBS.

Rename test_export_compile_execute_qnn_fb -> test_export_compile_execute_qnn_fbs for better readability

Typo in 'enviroment'

quic-rishinr · 2025-07-15T05:52:13Z

tests/cloud/test_infer.py

    )
+    check_infer(mocker=mocker, generation_len=20, **local_config)


Can we have a vlm qnn test as well?

I tried adding this test but got error, so removed from the test.

tests/cloud/test_infer.py

tests/transformers/models/test_causal_lm_models.py

Signed-off-by: Abukhoyer Shaik <[email protected]>

tests/transformers/models/test_causal_lm_models.py

Signed-off-by: Abukhoyer Shaik <[email protected]>

tests/cloud/test_export_compile_execute.py

tests/transformers/models/test_causal_lm_models.py

Signed-off-by: Abukhoyer Shaik <[email protected]>

tests/cloud/test_export_compile_execute.py

quic-rishinr · 2025-07-28T04:56:29Z

tests/transformers/models/test_causal_lm_models.py

+    "TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ",
+}
+
+extrenal_models = {"hpcai-tech/grok-1"}


nit: Typo "extrenal_models"

quic-rishinr · 2025-07-28T09:26:07Z

tests/transformers/models/test_causal_lm_models.py

+        else:
+            config_dict[config["model_name"]] = AutoConfig.for_model(
+                config["model_type"],
+                max_position_embeddings=config["max_position_embeddings"],


Can we use a get function instead of config[]? in case some params are not available it will throw a key error

tests/transformers/models/test_causal_lm_models.py

quic-rishinr · 2025-07-28T09:28:22Z

tests/transformers/models/test_causal_lm_models.py

+    config_objects = []
+    model_names = []
+    for name, config in model_config_dict.items():
+        if name in selected_model_names:


please make sure the selected_model_names is a set and not list or tuple

quic-rishinr · 2025-07-28T09:30:04Z

tests/transformers/models/test_causal_lm_models.py

+            trust_remote_code=model_name in extrenal_models,
+        )
+    else:
+        torch.manual_seed(42)


Can you define this at the top?

quic-rishinr · 2025-07-28T09:31:42Z

tests/transformers/models/test_causal_lm_models.py

-
+    if model_hf is None:
+        model_hf, _ = load_causal_lm_model(model_config)
+    model_hf_cb = copy.deepcopy(model_hf)


why we need to have a copy of model_hf for CB?

quic-rishinr · 2025-07-28T09:52:02Z

tests/transformers/models/test_causal_lm_models.py

    assert os.path.isfile(os.path.join(os.path.dirname(qpc_path), "qconfig.json"))


+# Load the custom models configuration data from the JSON file
+with open("tests/transformers/models/custom_tiny_model_configs.json", "r") as f:


is there a better way to handle this? We can use a class or using a global variable to load the configs and keep it. Read the configs and model names from each test methods

why we need to have a copy of model_hf for CB?
=> If we don't copy then we will have to load it again for CB testing as we cannot use the same model for CB and Non-CB testing.

Signed-off-by: Abukhoyer Shaik <[email protected]>

Adding dummy causal models for testing in regular CI run

87c253f

Signed-off-by: Abukhoyer Shaik <[email protected]>

abukhoy requested review from quic-rishinr, ochougul, quic-hemagnih and quic-amitraj as code owners May 29, 2025 11:42

abukhoy added 9 commits May 30, 2025 06:34

Test config modification

366a0f4

Signed-off-by: Abukhoyer Shaik <[email protected]>

modification

4c01d13

Signed-off-by: Abukhoyer Shaik <[email protected]>

remove randomness in pytorch output

c217780

Signed-off-by: Abukhoyer Shaik <[email protected]>

cloud json fixed

9b1d9f9

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

ed840d1

Signed-off-by: Abukhoyer Shaik <[email protected]>

Linter Fixed

e9812fc

Signed-off-by: Abukhoyer Shaik <[email protected]>

cloud tests changed

4f54c39

Signed-off-by: Abukhoyer Shaik <[email protected]>

CLI tests Single thread

1981c60

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

1aabae0

quic-rishinr added the 1.21.0 label Jun 24, 2025

abukhoy added 5 commits June 25, 2025 09:56

duplicate models are added

951c242

Signed-off-by: Abukhoyer Shaik <[email protected]>

QNN Cli test fixed

73e4ff9

Signed-off-by: Abukhoyer Shaik <[email protected]>

qnn config path disabled

d1cc91e

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

5a48f92

CLI tests are refactored

b47f518

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-amitraj assigned abukhoy Jul 8, 2025

quic-amitraj added the ready for review label Jul 8, 2025

abukhoy added 6 commits July 9, 2025 10:50

Merge branch 'main' into tests-optim

204592c

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

26e6469

Merge branch 'main' into tests-optim

eafe1d7

Merge branch 'main' into tests-optim

f6c10e6

Merge branch 'main' into tests-optim

3c6680d

Merge branch 'main' into tests-optim

d2a53ee

quic-rishinr requested changes Jul 15, 2025

View reviewed changes

comments are addressing

bdf96e4

Signed-off-by: Abukhoyer Shaik <[email protected]>

abukhoy added 7 commits July 16, 2025 11:50

comments are addressing

174d33e

Signed-off-by: Abukhoyer Shaik <[email protected]>

created dummy models config for fast testing

27a7b2f

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

f0cef88

comments are addressing

85f13ac

Signed-off-by: Abukhoyer Shaik <[email protected]>

comments are addressing

78a4dcf

Signed-off-by: Abukhoyer Shaik <[email protected]>

comments are addressing

af2f825

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

668ab2c

asmigosw reviewed Jul 22, 2025

View reviewed changes

tests/transformers/models/test_causal_lm_models.py Outdated Show resolved Hide resolved

comments addressing

d631fbd

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-hemagnih reviewed Jul 23, 2025

View reviewed changes

tests/cloud/test_export_compile_execute.py Show resolved Hide resolved

tests/cloud/test_export_compile_execute.py Show resolved Hide resolved

tests/transformers/models/test_causal_lm_models.py Show resolved Hide resolved

abukhoy added 3 commits July 25, 2025 10:10

swiftkv models added in the tests

6267b8b

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

7dee8e1

swiftkv models added in the tests

e328a04

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-rishinr requested changes Jul 28, 2025

View reviewed changes

causal_lm testing reformatted

3593e56

Signed-off-by: Abukhoyer Shaik <[email protected]>

		)
		check_infer(mocker=mocker, generation_len=20, **local_config)

[Tests]: Adding dummy causal models for testing in regular CI run #427

Are you sure you want to change the base?

[Tests]: Adding dummy causal models for testing in regular CI run #427

Conversation

abukhoy commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this PR:

What’s Changed:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abukhoy commented May 29, 2025 •

edited

Loading