Get to the start line with CIPHER

[gcapes]

• install cipher using instructions in the README

• get adam's pr branch to run - can remove this section which is only for windows:

  resources:
    any:
      shell_args:
        executable: pwsh.exe # required to get stdout/stderr redirects from docker command in a sensible encoding
  

This should then serve as a template for the rest of the tasks in hpcflow/matflow-new#403
Use the development branch of cipher-parse

[gcapes]

Installation didn't go smoothly, but I discovered that it will configure if I use python 3.11 instead of 3.13: micmog/CIPHER#14

I also removed all the MKL options as I don't already have that or an intel compiler, and as far as I recall MKL is an intel library for faster maths operations, and CIPHER will use LAPACK if MKL is not available. So the configure command then became:

./configure --download-metis --download-parmetis --download-chaco --download-triangle --download-ctetgen --download-pragmatic --download-eigen --download-hypre --download-ml --download-hdf5 --download-zlib --download-yaml --download-p4est --with-pthread  --with-cxx-dialect=C++11 --with-debugging=0  PETSC_ARCH=cipher PETSC_DIR=$CIPHER_DIR/petsc --download-fblaslapack=1

Then make PETSC_DIR=/home/mbexegc2/software/cipher/petsc PETSC_ARCH=cipher all

[gcapes]

It looks to have installed correctly, with the make check results passing.

[gcapes]

The GrainBoundaryPrecipitate.yaml example looked to run ok too (from the README) - it ran for maybe 5-10 mins without errors before I cancelled it.

[gcapes]

@aplowman Could you please share your cipher processing env? I get this error currently:

The environment 'cipher_processing_env' is not defined on this machine, so the submission cannot be created

[gcapes]

Oh, I guess it's a python env with cipher-parse in it?

[aplowman]

Yeah, just Python with cipher-parse.

[gcapes]

cipher parse needs python <3.11 🙄
Installing python 3.10.18 ...

[gcapes]

@aplowman
I think I've run into a problem.
I'm trying to install matflow into the same venv as used for cipher_processing_env. Because it needs it to run the workflow and complains if I only have cipher-parse in the venv. But there's a conflicting requirement

$ pip install ./matflow-new/ # i.e. from my locally cloned repo, on the `softaware/cipher` branch.
.
.
.
Building wheels for collected packages: matflow-new
  Building wheel for matflow-new (pyproject.toml) ... done
  Created wheel for matflow-new: filename=matflow_new-0.3.0a176-py3-none-any.whl size=154383 sha256=7abef898b23ae123ad86555faae7dcfa87ee20571d84954bdfed15d5a3d705d3
  Stored in directory: /home/mbexegc2/.cache/pip/wheels/fd/72/40/aab226ed6a595770ed9e14ce69f961650a6f272b380a97a80e
Successfully built matflow-new
Installing collected packages: watchdog, termcolor, ruamel-yaml, reretry, pyzmq, numcodecs, fsspec, colorama, click, bcrypt, zarr, valida, pynacl, cryptography, paramiko, hpcflow-new2, matflow-new
  Attempting uninstall: ruamel-yaml
    Found existing installation: ruamel.yaml 0.17.40
    Uninstalling ruamel.yaml-0.17.40:
      Successfully uninstalled ruamel.yaml-0.17.40
  Attempting uninstall: pyzmq
    Found existing installation: pyzmq 27.0.1
    Uninstalling pyzmq-27.0.1:
      Successfully uninstalled pyzmq-27.0.1
  Attempting uninstall: numcodecs
    Found existing installation: numcodecs 0.13.1
    Uninstalling numcodecs-0.13.1:
      Successfully uninstalled numcodecs-0.13.1
  Attempting uninstall: zarr
    Found existing installation: zarr 2.18.3
    Uninstalling zarr-2.18.3:
      Successfully uninstalled zarr-2.18.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cipher-parse 0.6.0a1 requires ruamel.yaml<0.18.0,>=0.17.21, but you have ruamel-yaml 0.18.14 which is incompatible.
Successfully installed bcrypt-4.3.0 click-8.2.1 colorama-0.4.6 cryptography-45.0.6 fsspec-2025.7.0 hpcflow-new2-0.2.0a223 matflow-new-0.3.0a176 numcodecs-0.12.1 paramiko-3.5.1 pynacl-1.5.0 pyzmq-26.4.0 reretry-0.11.8 ruamel-yaml-0.18.14 termcolor-2.5.0 valida-0.7.5 watchdog-6.0.0 zarr-2.17.2

I assume this means it won't work (just about to test that).

$ pip list
Package                   Version
------------------------- --------------
aiohappyeyeballs          2.6.1
aiohttp                   3.12.15
aiosignal                 1.4.0
anyio                     4.10.0
argon2-cffi               25.1.0
argon2-cffi-bindings      25.1.0
arrow                     1.3.0
asciitree                 0.3.3
asttokens                 3.0.0
async-timeout             5.0.1
attrs                     25.3.0
bcrypt                    4.3.0
beautifulsoup4            4.13.4
bleach                    6.2.0
certifi                   2025.8.3
cffi                      1.17.1
charset-normalizer        3.4.2
cipher-parse              0.6.0a1
click                     8.2.1
cmocean                   4.0.3
colorama                  0.4.6
colorcet                  3.1.0
comm                      0.2.3
contourpy                 1.3.2
cryptography              45.0.6
cycler                    0.12.1
damask                    3.0.0a7.post0
debugpy                   1.8.16
decorator                 5.2.1
DefDAP                    0.93.6
defusedxml                0.7.1
exceptiongroup            1.3.0
executing                 2.2.0
fasteners                 0.19
fastjsonschema            2.21.1
fonttools                 4.59.0
fqdn                      1.5.1
frozenlist                1.7.0
fsspec                    2025.7.0
h5py                      3.14.0
hpcflow-new2              0.2.0a223
idna                      3.10
imageio                   2.37.0
ipydatawidgets            4.3.2
ipykernel                 6.30.1
ipympl                    0.9.7
ipython                   8.21.0
ipython-genutils          0.2.0
ipywidgets                7.8.5
isoduration               20.11.0
itk-core                  5.4.4.post1
itk-filtering             5.4.4.post1
itk-meshtopolydata        0.11.1
itk-numerics              5.4.4.post1
itkwidgets                0.32.6
jedi                      0.19.2
Jinja2                    3.1.6
jsonpointer               3.0.0
jsonschema                4.25.0
jsonschema-specifications 2025.4.1
jupyter_client            8.6.3
jupyter_core              5.8.1
jupyter-events            0.12.0
jupyter_server            2.16.0
jupyter_server_proxy      4.4.0
jupyter_server_terminals  0.5.3
jupyterlab_pygments       0.3.0
jupyterlab_widgets        1.1.11
kiwisolver                1.4.8
lark                      1.2.2
lazy_loader               0.4
markdown-it-py            3.0.0
MarkupSafe                3.0.2
matflow-new               0.3.0a176
matplotlib                3.10.5
matplotlib-inline         0.1.7
matplotlib-scalebar       0.9.0
mdurl                     0.1.2
meshio                    5.3.5
mistune                   3.1.3
more-itertools            10.7.0
mpld3                     0.5.11
msgpack                   1.1.1
multidict                 6.6.3
nbclient                  0.10.2
nbconvert                 7.16.6
nbformat                  5.10.4
nest-asyncio              1.6.0
networkx                  3.4.2
notebook                  6.4.13
numcodecs                 0.12.1
numpy                     1.26.4
overrides                 7.7.0
packaging                 25.0
pandas                    1.5.3
pandocfilters             1.5.1
paramiko                  3.5.1
parse                     1.20.2
parso                     0.8.4
PeakUtils                 1.3.5
pexpect                   4.9.0
pillow                    11.3.0
pillow-avif-plugin        1.5.2
pip                       23.0.1
platformdirs              4.3.8
plotly                    5.24.1
pooch                     1.8.2
prometheus_client         0.22.1
prompt_toolkit            3.0.51
propcache                 0.3.2
psutil                    7.0.0
ptyprocess                0.7.0
pure_eval                 0.2.3
pycparser                 2.22
Pygments                  2.19.2
PyNaCl                    1.5.0
pyparsing                 3.2.3
python-dateutil           2.9.0.post0
python-json-logger        3.3.0
pytz                      2025.2
pyvista                   0.43.10
PyYAML                    6.0.2
pyzmq                     26.4.0
referencing               0.36.2
requests                  2.32.4
reretry                   0.11.8
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rfc3987-syntax            1.1.0
rich                      14.1.0
rpds-py                   0.27.0
ruamel.yaml               0.18.14
ruamel.yaml.clib          0.2.12
scikit-image              0.25.2
scipy                     1.15.3
scooby                    0.10.1
Send2Trash                1.8.3
setuptools                65.5.0
simpervisor               1.0.0
six                       1.17.0
sniffio                   1.3.1
soupsieve                 2.7
stack-data                0.6.3
tenacity                  9.1.2
termcolor                 2.5.0
terminado                 0.18.1
tifffile                  2025.5.10
tinycss2                  1.4.0
tornado                   6.5.1
traitlets                 5.6.0
traittypes                0.2.1
trame                     2.5.2
trame-client              3.9.2
trame-common              1.0.1
trame-components          2.5.0
trame-deckgl              2.0.3
trame-markdown            2.0.2
trame-matplotlib          2.0.3
trame-plotly              2.1.1
trame-rca                 2.1.3
trame-router              2.3.0
trame-server              3.6.0
trame-simput              2.6.0
trame-vega                2.1.1
trame-vtk                 2.9.1
trame-vuetify             2.9.0
types-python-dateutil     2.9.0.20250708
typing_extensions         4.14.1
uri-template              1.3.0
urllib3                   2.5.0
valida                    0.7.5
vecmaths                  0.1.6
vtk                       9.5.0
watchdog                  6.0.0
wcwidth                   0.2.13
webcolors                 24.11.1
webencodings              0.5.1
websocket-client          1.8.0
widgetsnbextension        3.6.10
wslink                    2.3.4
yarl                      1.20.1
zarr                      2.17.2
zstandard                 0.23.0

Ok, so the first task runs, but I get this error in /home/mbexegc2/projects/lightform/cipher_2025-08-08_195516/artifacts/submissions/0/js_std/0/js_0_std.log

Tessellating regions...done!
Finding neighbouring voxels...done!
Finding neighbour list...done!
Finding interface map matrix...done!
Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/cipher_2025-08-08_195516/artifacts/submissions/0/scripts/simulate_grain_growth_act_0.py", line 44, in <module>
    write_cipher_input(**func_kwargs)
  File "/home/mbexegc2/projects/lightform/cipher_2025-08-08_195516/artifacts/submissions/0/scripts/simulate_grain_growth_act_0.py", line 7, in write_cipher_input
    inp.geometry.write_VTK("initial.vti")
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/cipher_parse/geometry.py", line 1097, in write_VTK
    grid.save(path)
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/pyvista/core/dataobject.py", line 147, in save
    self._store_metadata()
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/pyvista/core/dataobject.py", line 173, in _store_metadata
    assoc_data = getattr(self, f'_association_{assoc_name}_names')
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/pyvista/core/dataset.py", line 150, in __getattr__
    return super().__getattribute__(item)
AttributeError: 'ImageData' object has no attribute '_association_bitarray_names'

Now whether the two are related I don't know, but I'm not at the start line yet.

[aplowman]

It should still work despite the pip incompatibility (looks like ruamel.yaml just needs bumping in cipher-parse).

Comparing your packages to mine, I have pyvista 0.45.2 vs your 0.43.10. Comparing Pyvista's dependencies (0.43.10 and 0.45.2), I can see the VTK dependency is restricted in 0.45.2 to a version that excludes the version your have. So that could be the issue. Try upgrading pyvista to the latest, which should downgrade VTK. Good luck!

[gcapes]

Using latest pyvista (0.46.0), or indeed 0.45.2 I get this error in artifacts/submissions/0/app_std/3.txt

Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/hpcflow/sdk/core/utils.py", line 1186, in redirect_std_to_file
    yield
  File "/home/mbexegc2/projects/lightform/cipher_2025-08-11_101146/artifacts/submissions/0/scripts/simulate_grain_growth_act_2.py", line 66, in <module>
    func_kwargs = EAR.get_py_script_func_kwargs(
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/hpcflow/sdk/core/actions.py", line 930, in get_py_script_func_kwargs
    kwargs.update(self.get_OFP_output_files())
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/hpcflow/sdk/core/actions.py", line 848, in get_OFP_output_files
    return {
  File "/home/mbexegc2/projects/lightform/.venv-cipher_processing/lib/python3.10/site-packages/hpcflow/sdk/core/actions.py", line 849, in <dictcomp>
    file_spec.label: Path(cast("str", file_spec.name.value()))
  File "/home/mbexegc2/software/python/3.10.18/lib/python3.10/pathlib.py", line 960, in __new__
    self = cls._from_parts(args)
  File "/home/mbexegc2/software/python/3.10.18/lib/python3.10/pathlib.py", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/home/mbexegc2/software/python/3.10.18/lib/python3.10/pathlib.py", line 578, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not list

This is my latest pip list used, which gave the error above:

Package                   Version
------------------------- --------------
aiohappyeyeballs          2.6.1
aiohttp                   3.12.15
aiosignal                 1.4.0
anyio                     4.10.0
argon2-cffi               25.1.0
argon2-cffi-bindings      25.1.0
arrow                     1.3.0
asciitree                 0.3.3
asttokens                 3.0.0
async-timeout             5.0.1
attrs                     25.3.0
bcrypt                    4.3.0
beautifulsoup4            4.13.4
bleach                    6.2.0
certifi                   2025.8.3
cffi                      1.17.1
charset-normalizer        3.4.2
cipher-parse              0.6.0a1
click                     8.2.1
cmocean                   4.0.3
colorama                  0.4.6
colorcet                  3.1.0
comm                      0.2.3
contourpy                 1.3.2
cryptography              45.0.6
cycler                    0.12.1
damask                    3.0.0a7.post0
debugpy                   1.8.16
decorator                 5.2.1
DefDAP                    0.93.6
defusedxml                0.7.1
exceptiongroup            1.3.0
executing                 2.2.0
fasteners                 0.19
fastjsonschema            2.21.1
fonttools                 4.59.0
fqdn                      1.5.1
frozenlist                1.7.0
fsspec                    2025.7.0
h5py                      3.14.0
hpcflow-new2              0.2.0a223
idna                      3.10
imageio                   2.37.0
ipydatawidgets            4.3.2
ipykernel                 6.30.1
ipympl                    0.9.7
ipython                   8.21.0
ipython-genutils          0.2.0
ipywidgets                7.8.5
isoduration               20.11.0
itk-core                  5.4.4.post1
itk-filtering             5.4.4.post1
itk-meshtopolydata        0.11.1
itk-numerics              5.4.4.post1
itkwidgets                0.32.6
jedi                      0.19.2
Jinja2                    3.1.6
jsonpointer               3.0.0
jsonschema                4.25.0
jsonschema-specifications 2025.4.1
jupyter_client            8.6.3
jupyter_core              5.8.1
jupyter-events            0.12.0
jupyter_server            2.16.0
jupyter_server_proxy      4.4.0
jupyter_server_terminals  0.5.3
jupyterlab_pygments       0.3.0
jupyterlab_widgets        1.1.11
kiwisolver                1.4.8
lark                      1.2.2
lazy_loader               0.4
markdown-it-py            3.0.0
MarkupSafe                3.0.2
matflow-new               0.3.0a176
matplotlib                3.10.5
matplotlib-inline         0.1.7
matplotlib-scalebar       0.9.0
mdurl                     0.1.2
meshio                    5.3.5
mistune                   3.1.3
more-itertools            10.7.0
mpld3                     0.5.11
msgpack                   1.1.1
multidict                 6.6.3
nbclient                  0.10.2
nbconvert                 7.16.6
nbformat                  5.10.4
nest-asyncio              1.6.0
networkx                  3.4.2
notebook                  6.4.13
numcodecs                 0.12.1
numpy                     1.26.4
overrides                 7.7.0
packaging                 25.0
pandas                    1.5.3
pandocfilters             1.5.1
paramiko                  3.5.1
parse                     1.20.2
parso                     0.8.4
PeakUtils                 1.3.5
pexpect                   4.9.0
pillow                    11.3.0
pillow-avif-plugin        1.5.2
pip                       23.0.1
platformdirs              4.3.8
plotly                    5.24.1
pooch                     1.8.2
prometheus_client         0.22.1
prompt_toolkit            3.0.51
propcache                 0.3.2
psutil                    7.0.0
ptyprocess                0.7.0
pure_eval                 0.2.3
pycparser                 2.22
Pygments                  2.19.2
PyNaCl                    1.5.0
pyparsing                 3.2.3
python-dateutil           2.9.0.post0
python-json-logger        3.3.0
pytz                      2025.2
pyvista                   0.45.2
PyYAML                    6.0.2
pyzmq                     26.4.0
referencing               0.36.2
requests                  2.32.4
reretry                   0.11.8
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rfc3987-syntax            1.1.0
rich                      14.1.0
rpds-py                   0.27.0
ruamel.yaml               0.18.14
ruamel.yaml.clib          0.2.12
scikit-image              0.25.2
scipy                     1.15.3
scooby                    0.10.1
Send2Trash                1.8.3
setuptools                65.5.0
simpervisor               1.0.0
six                       1.17.0
sniffio                   1.3.1
soupsieve                 2.7
stack-data                0.6.3
tenacity                  9.1.2
termcolor                 2.5.0
terminado                 0.18.1
tifffile                  2025.5.10
tinycss2                  1.4.0
tornado                   6.5.1
traitlets                 5.6.0
traittypes                0.2.1
trame                     2.5.2
trame-client              3.9.2
trame-common              1.0.1
trame-components          2.5.0
trame-deckgl              2.0.3
trame-markdown            2.0.2
trame-matplotlib          2.0.3
trame-plotly              2.1.1
trame-rca                 2.1.3
trame-router              2.3.0
trame-server              3.6.0
trame-simput              2.6.0
trame-vega                2.1.1
trame-vtk                 2.9.1
trame-vuetify             2.9.0
types-python-dateutil     2.9.0.20250708
typing_extensions         4.14.1
uri-template              1.3.0
urllib3                   2.5.0
valida                    0.7.5
vecmaths                  0.1.6
vtk                       9.4.2
watchdog                  6.0.0
wcwidth                   0.2.13
webcolors                 24.11.1
webencodings              0.5.1
websocket-client          1.8.0
widgetsnbextension        3.6.10
wslink                    2.3.4
yarl                      1.20.1
zarr                      2.17.2
zstandard                 0.23.0

[aplowman]

Can you try updaing hpcflow? I think this is something I fixed a few weeks ago:

hpcflow/hpcflow-new@5f83222

[gcapes]

Great, thanks. I guess my hpcflow was determined by the matflow release before your PR branch. Anyway, it's working now, thanks!