Onnx model testing v2 #39

oivie · 2025-08-13T18:21:34Z

Description

Created multiple unconventional neural network models to stress-test the JSTProve system and identify integration limitations. The models successfully test different failure modes in the PyTorch → ONNX → JSTProve pipeline, discovering actual system limitations including unsupported operations (Tanh/Sigmoid) and ONNX export edge cases.

Added three ONNX models with varying levels of architectural weirdness:

model_redundant.onnx: Simple model with redundant ReLU layers (baseline)
model_wrong_dimension.onnx: Mixes 1D/2D operations, uses unsupported activations
model_supported_weird.onnx: Architecturally chaotic but uses only supported operations

Related Issue

Testing framework validation for edge cases and unusual model architectures
Integration testing between PyTorch, ONNX, and JSTProve systems

Key Findings

JSTProve Operation Limitations Discovered: Unsupported op type: {'Tanh', 'Sigmoid'}
ONNX Export Limitations Confirmed
Models using AdaptiveAvgPool2d with incompatible dimensions fail ONNX export as expected.

Integration Framework Validation

All models successfully integrate with pytest framework
Test discovery and execution pipeline functions correctly
Proper error handling and reporting for different failure modes

Note:
Expected behavior: model_redundant and model_supported_weird should pass, model_wrong_dimension will fail at JSTProve compilation due to unsupported operations

- Add model_redundant.onnx: Simple model with redundant layers - Add model_wrong_dimension.onnx: Model mixing 1D/2D operations - Add model_supported_weird.onnx: Architecturally weird but uses only supported ops - Add generation scripts for reproducing models - Add test file for model_redundant Models test different failure modes: - ONNX export limitations - JSTProve operation support gaps (Tanh/Sigmoid unsupported) - Integration framework edge cases

HudsonGraeme

Looks 🔥!!
Left some small comments mostly about paths and newlines 😄

HudsonGraeme · 2025-08-14T04:59:20Z

python/scripts/export_wrong_dimension.py

+import os
+
+# Add the models directory to Python path
+sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')


Preference to use a relative abs path when possible; this will allow anyone who pulls the repo to easily run the scripts without needing to meddle with the path :)

Suggested change

sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx')))

HudsonGraeme · 2025-08-14T05:02:15Z

python/scripts/export_wrong_dimension.py

+except Exception as e:
+    print(f"❌ Error: {e}")
+    import traceback
+    traceback.print_exc()


If you open this file in the files tab you may note this do not enter sign; it indicates the file isn't POSIX compliant due to the lack of newline at the end. This is not really a big deal since we're no longer in the 19th century but still preferable to include a newline at the end if possible :)

HudsonGraeme · 2025-08-14T05:04:24Z

python/scripts/generate_io_conv2d.py

+    input_tensor = torch.randn(1, 1, 28, 28)
+    input_numpy = input_tensor.numpy()
+
+    session = ort.InferenceSession("/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx/model_conv2d.onnx")


Suggested change

session = ort.InferenceSession("/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx/model_conv2d.onnx")

session = ort.InferenceSession(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx', 'model_conv2d.onnx')))

HudsonGraeme · 2025-08-14T05:04:36Z

python/scripts/generate_io_redundant.py

+import sys
+
+# Add the models directory to Python path
+sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')


Suggested change

sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx'))

HudsonGraeme · 2025-08-14T05:05:16Z

python/scripts/generate_io_wrong_dimension.py

+import sys
+
+# Add the models directory to Python path
+sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')


Suggested change

sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx'))

HudsonGraeme · 2025-08-14T05:13:08Z

python/scripts/generate_io_wrong_dimension.py

+    output_data = output.detach().numpy().tolist()
+
+    print("Creating output directory...")
+    io_dir = "python/testing/core/input_output_data/model_wrong_dimension/"


Suggested change

io_dir = "python/testing/core/input_output_data/model_wrong_dimension/"

io_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'testing', 'core', 'input_output_data', 'model_wrong_dimension')

HudsonGraeme · 2025-08-14T05:13:47Z

python/testing/core/tests/test_conv2d_model.py

+
+def test_conv2d_model_output():
+    # Paths - using relative paths since files are in the same directory as the test
+    onnx_model_path = "python/models/models_onnx/model_conv2d.onnx"


Suggested change

onnx_model_path = "python/models/models_onnx/model_conv2d.onnx"

onnx_model_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx', 'model_conv2d.onnx')

HudsonGraeme · 2025-08-14T05:14:27Z

python/testing/core/tests/test_model_redundant.py

+import sys
+
+# Add the models directory to Python path
+sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')


Suggested change

sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')

sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx')))

HudsonGraeme · 2025-08-14T05:15:13Z

python/testing/core/tests/test_model_redundant.py

+
+try:
+    print("Importing RedundantModel...")
+    from redundant_layers_model import RedundantModel


Hmm this one might be left out of the commit too or potentially I missed it somehow again too

HudsonGraeme · 2025-08-14T05:15:36Z

python/testing/core/tests/test_model_redundant.py

+
+    print("Creating output directory...")
+    # Save input/output
+    io_dir = "python/testing/core/input_output_data/model_redundant/"


Suggested change

io_dir = "python/testing/core/input_output_data/model_redundant/"

io_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'testing', 'core', 'input_output_data', 'model_redundant')

…SIX compliance

Elena Pashkova added 2 commits July 24, 2025 17:26

Add Conv2D ONNX model, IO script, and test

a0eafe2

HudsonGraeme reviewed Aug 14, 2025

View reviewed changes

Fix relative path in export_wrong_dimension.py and add newline for PO…

4f1ea60

…SIX compliance

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Onnx model testing v2 #39

Onnx model testing v2 #39

Uh oh!

oivie commented Aug 13, 2025

Uh oh!

HudsonGraeme left a comment

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

HudsonGraeme Aug 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	sys.path.append('/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx')
	sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx')))

	session = ort.InferenceSession("/Users/elenapashkova/GravyTesting-Internal/python/models/models_onnx/model_conv2d.onnx")
	session = ort.InferenceSession(os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx', 'model_conv2d.onnx')))

	io_dir = "python/testing/core/input_output_data/model_wrong_dimension/"
	io_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'testing', 'core', 'input_output_data', 'model_wrong_dimension')

	onnx_model_path = "python/models/models_onnx/model_conv2d.onnx"
	onnx_model_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'models', 'models_onnx', 'model_conv2d.onnx')

Onnx model testing v2 #39

Are you sure you want to change the base?

Onnx model testing v2 #39

Uh oh!

Conversation

oivie commented Aug 13, 2025

Description

Related Issue

Key Findings

Integration Framework Validation

Uh oh!

HudsonGraeme left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants