🐛[BUG]: Install CI Failing

[coreyjadams]

Version

Main

On which installation method(s) does this occur?

Pip

Describe the issue

Install CI is failing across a broad suite of tests. Nearly half the runs fall over on one test, from strata. Additionally, Mesh compile tests appear to be failing on just Ubuntu + Pip. So, paging both Peter H and Peter S for triage?

https://github.com/NVIDIA/physicsnemo/actions/runs/29723676624/job/88291860303

Minimum reproducible example

Relevant log output

Environment details

[coreyjadams]

Strata error:

=========================== short test summary info ============================
FAILED test/experimental/models/strata/test_strata.py::test_non_regression[strata_bilinear_pd2] - assert False
 +  where False = <built-in method allclose of type object at 0xffdad599d680>(tensor([[[[[-4.5489e+00, -5.3457e+00, -4.4345e+00,  ..., -4.1807e+00,\n            -4.8146e+00, -6.0303e+00],\n         ...e+00],\n           [ 4.8798e+00,  2.6211e+00,  3.3874e+00,  ...,  1.5551e+00,\n             1.7964e+00, -1.8749e+00]]]]]), tensor([[[[[-4.5489e+00, -5.3457e+00, -4.4345e+00,  ..., -4.1807e+00,\n            -4.8146e+00, -6.0308e+00],\n         ...e+00],\n           [ 4.8798e+00,  2.6210e+00,  3.3874e+00,  ...,  1.5550e+00,\n             1.7963e+00, -1.8749e+00]]]]]), atol=0.002, rtol=0.002)
 +    where <built-in method allclose of type object at 0xffdad599d680> = torch.allclose

[coreyjadams]

Mesh compile error:

FAILED test/mesh/mesh/test_compile.py::test_cache_rebuild_paths_under_fullgraph_compile - torch._dynamo.exc.Unsupported: Observed exception
  Explanation: Dynamo found no exception handler at the top-level compiled function when encountering an exception. Exception will propagate outside the compiled region.
  Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the `torch.compile` call, or by using `torch.compiler.set_stance("force_eager")`. 
  Hint: It may be possible to write Dynamo tracing rules for this code. Please report an issue to PyTorch if you encounter this graph break often and it is causing performance issues.

  Developer debug context: raised exception AttributeError("Cannot set attribute _non_tensordict in GenericContextWrappingVariable(Mesh) as this entry is not amongst the expected ones ({'cells', 'point_data', 'global_data', '_cache', 'cell_data', 'points'}).")

 For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0088.html

from user code:
   File "/home/runner/work/physicsnemo/physicsnemo/test/mesh/mesh/test_compile.py", line 189, in fn
    mesh = Mesh(points=p, cells=c)
  File "/opt/hostedtoolcache/Python/3.13.14/x64/lib/python3.13/site-packages/torch/_dynamo/polyfills/__init__.py", line 513, in instantiate_user_defined_class_object
    obj.__init__(*args, **kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.13.14/x64/lib/python3.13/site-packages/tensordict/tensorclass.py", line 1388, in wrapper
    self.__setattr_parent__("_non_tensordict", {})
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.13.14/x64/lib/python3.13/site-packages/tensordict/tensorclass.py", line 1949, in _setattr_tensor_only
    raise AttributeError(
    ^

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

[pzharrington]

Strata error should be fixed by #1858 . @peterdsharpe let me know if you want to just push your mesh fix(es) there as well

[peterdsharpe]

@pzharrington will do, thanks for opening the PR!

Tracing the issue now, initial findings on the compile regression are that it's caused by torch 2.13 (rather than 2.12 before, which passed). Root-causing further...

[peterdsharpe]

Ok, the torch.compile issue is due to PyTorch 2.13 (released July 8) shipping breaking changes that clash with TesnorDict 0.13.0. TensorDict has implemented a fix, but they haven't yet cut a release. Opening an issue on TensorDict for this.

Strategy is going to be to pin torch<2.13 until pytorch/tensordict#1748 is fixed (which should be soon, the fix is basically a straightforward "cut a release")