You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Installing netCDF4 (GitHub version with commit f9dc8b2) inside a virtual environment using pip install causes an error complaining about the lack of mpi4py package. This error is thrown at the user regardless whether the mpi4py is installed within the same virtual environment or not. This is not limited to this version of netcdf4-python, as a few latest releases that I have tried, also demonstrate the same behaviour. Here is an example:
(test2) [netcdf4-python]$ pip install --no-binary mpi4py --no-cache mpi4pyLooking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2020/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2020/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/genericCollecting mpi4py Downloading mpi4py-3.1.5.tar.gz (2.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 68.8 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... doneBuilding wheels for collected packages: mpi4py Building wheel for mpi4py (pyproject.toml) ... done Created wheel for mpi4py: filename=mpi4py-3.1.5-cp310-cp310-linux_x86_64.whl size=3365058 sha256=114d79ee629f43bafeb69a273c25984db3a644dd64ea57b47c84c7a058c0b7a9 Stored in directory: /tmp/pip-ephem-wheel-cache-o8sr3xtk/wheels/18/2b/7f/c852523089e9182b45fca50ff56f49a51eeb6284fd25a66713Successfully built mpi4pyInstalling collected packages: mpi4pySuccessfully installed mpi4py-3.1.5(test2) [netcdf4-python]$ pip install -v .Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2020/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo2020/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/genericProcessing /home/user/github-repos/netcdf4-python Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [28 lines of output] -L/home/user/.local/lib64 -lnetcdf -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/lib -lhdf5 -lsz reading from setup.cfg... using nc-config... checking /home/user/.local/include... hdf5 headers not found in /home/user/.local/include nc-config did provide path to HDF5 headers, search standard locations... checking /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/include... HDF5 library version: 1.12.1 headers found in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/include HDF5 library version: 1.12.1 found in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1 using netcdf library version b'4.9.3-development' using Cython to compile netCDF4.pyx... netcdf lib has parallel functions Traceback (most recent call last): File "/home/user/virtual-envs/test2/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> main() File "/home/user/virtual-envs/test2/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/home/user/virtual-envs/test2/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel return hook(config_settings) File "/tmp/pip-build-env-ifjbt4w7/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "/tmp/pip-build-env-ifjbt4w7/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires self.run_setup() File "/tmp/pip-build-env-ifjbt4w7/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup exec(code, locals()) File "<string>", line 395, in <module> ModuleNotFoundError: No module named 'mpi4py' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip.error: subprocess-exited-with-error× Getting requirements to build wheel did not run successfully.│ exit code: 1╰─> See above for output.note: This error originates from a subprocess, and is likely not a problem with pip.(test2) [netcdf4-python]$ pythonPython 3.10.2 (main, Feb 4 2022, 19:10:35) [GCC 9.3.0] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import mpi4py>>> mpi4py.__file__'/home/user/virtual-envs/test2/lib/python3.10/site-packages/mpi4py/__init__.py'
However, if the installation proceeds with using the setup.py, it works smoothly and there won't be any problem:
(test2) [netcdf4-python]$ python -m setup installreading from setup.cfg...-L/home/user/.local/lib64 -lnetcdf-L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/lib -lhdf5 -lszusing nc-config...checking /home/user/.local/include...hdf5 headers not found in /home/user/.local/includenc-config did provide path to HDF5 headers, search standard locations...checking /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/include...HDF5 library version: 1.12.1 headers found in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/includeHDF5 library version: 1.12.1 found in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1using netcdf library version b'4.9.3-development'using Cython to compile netCDF4.pyx...netcdf lib has parallel functionsNETCDF_PLUGIN_DIR not set, no netcdf compression plugins installedrunning install/home/user/virtual-envs/test2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.!! ******************************************************************************** Please avoid running ``setup.py`` directly. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. ********************************************************************************!! self.initialize_options()/home/user/virtual-envs/test2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.!! ******************************************************************************** Please avoid running ``setup.py`` and ``easy_install``. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://github.com/pypa/setuptools/issues/917 for details. ********************************************************************************!! self.initialize_options()running bdist_eggrunning egg_infowriting src/netCDF4.egg-info/PKG-INFOwriting dependency_links to src/netCDF4.egg-info/dependency_links.txtwriting entry points to src/netCDF4.egg-info/entry_points.txtwriting requirements to src/netCDF4.egg-info/requires.txtwriting top-level names to src/netCDF4.egg-info/top_level.txtreading manifest file 'src/netCDF4.egg-info/SOURCES.txt'reading manifest template 'MANIFEST.in'warning: no previously-included files found matching 'examples/data'warning: no previously-included files found matching 'src/netCDF4/_netCDF4.c'adding license file 'LICENSE'writing manifest file 'src/netCDF4.egg-info/SOURCES.txt'installing library code to build/bdist.linux-x86_64/eggrunning install_librunning build_pyrunning build_extCompiling src/netCDF4/_netCDF4.pyx because it changed.[1/1] Cythonizing src/netCDF4/_netCDF4.pyxbuilding 'netCDF4._netCDF4' extensiongcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/user/.local/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/include -I/home/user/virtual-envs/test2/lib/python3.10/site-packages/numpy/core/include -I/home/user/virtual-envs/test2/lib/python3.10/site-packages/mpi4py/include -Iinclude -I/home/user/virtual-envs/test2/include -I/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.10.2/include/python3.10 -c src/netCDF4/_netCDF4.c -o build/temp.linux-x86_64-cpython-310/src/netCDF4/_netCDF4.osrc/netCDF4/_netCDF4.c: In function ‘__pyx_pf_7netCDF4_8_netCDF4_7_MemBuf_4__dealloc__’:src/netCDF4/_netCDF4.c:13413:20: warning: passing argument 1 of ‘free’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]13413 | free(__pyx_v_self->memory); | ~~~~~~~~~~~~^~~~~~~~In file included from /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.10.2/include/python3.10/Python.h:34, from src/netCDF4/_netCDF4.c:65:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/include/stdlib.h:565:25: note: expected ‘void *’ but argument is of type ‘const void *’ 565 | extern void free (void *__ptr) __THROW; | ~~~~~~^~~~~src/netCDF4/_netCDF4.c: In function ‘__pyx_f_7netCDF4_8_netCDF4__set_att’:src/netCDF4/_netCDF4.c:18963:106: warning: passing argument 5 of ‘nc_put_att_string’ from incompatible pointer type [-Wincompatible-pointer-types]18963 | __pyx_v_ierr = nc_put_att_string(__pyx_v_grpid, __pyx_v_varid, __pyx_v_attname, __pyx_v_N, __pyx_v_string_ptrs); | ^~~~~~~~~~~~~~~~~~~ | | | char **In file included from src/netCDF4/_netCDF4.c:1249:/home/user/.local/include/netcdf.h:1091:44: note: expected ‘const char **’ but argument is of type ‘char **’ 1091 | size_t len, const char **op); | ~~~~~~~~~~~~~^~src/netCDF4/_netCDF4.c:19490:103: warning: passing argument 5 of ‘nc_put_att_string’ from incompatible pointer type [-Wincompatible-pointer-types]19490 | __pyx_v_ierr = nc_put_att_string(__pyx_v_grpid, __pyx_v_varid, __pyx_v_attname, 1, (&__pyx_v_datstring)); | ~^~~~~~~~~~~~~~~~~~~ | | | char **In file included from src/netCDF4/_netCDF4.c:1249:/home/user/.local/include/netcdf.h:1091:44: note: expected ‘const char **’ but argument is of type ‘char **’ 1091 | size_t len, const char **op); | ~~~~~~~~~~~~~^~src/netCDF4/_netCDF4.c: In function ‘__pyx_f_7netCDF4_8_netCDF4__read_compound’:src/netCDF4/_netCDF4.c:85513:35: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]85513 | for (__pyx_v_nf = 0; __pyx_v_nf < __pyx_t_7; __pyx_v_nf++) { | ^src/netCDF4/_netCDF4.c: In function ‘__pyx_f_7netCDF4_8_netCDF4__read_enum’:src/netCDF4/_netCDF4.c:89720:39: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]89720 | for (__pyx_v_nmem = 0; __pyx_v_nmem < __pyx_t_11; __pyx_v_nmem++) { | ^src/netCDF4/_netCDF4.c: In function ‘__pyx_pw_7netCDF4_8_netCDF4_8Variable_1__init__’:src/netCDF4/_netCDF4.c:49807:28: warning: ‘__pyx_v_dimids’ may be used uninitialized in this function [-Wmaybe-uninitialized]49807 | __pyx_v_ierr = nc_def_var(__pyx_v_self->_grpid, __pyx_v_varname, __pyx_v_xtype, __pyx_v_ndims, __pyx_v_dimids, (&__pyx_v_self->_varid)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:48022:8: note: ‘__pyx_v_dimids’ was declared here48022 | int *__pyx_v_dimids; | ^~~~~~~~~~~~~~src/netCDF4/_netCDF4.c: In function ‘__pyx_f_7netCDF4_8_netCDF4__get_types’:src/netCDF4/_netCDF4.c:19952:12: warning: ‘__pyx_v_typeids’ may be used uninitialized in this function [-Wmaybe-uninitialized]19952 | nc_type *__pyx_v_typeids; | ^~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c: In function ‘__pyx_pw_7netCDF4_8_netCDF4_7Dataset_1__init__’:src/netCDF4/_netCDF4.c:25956:30: warning: ‘__pyx_v_mpiinfo’ may be used uninitialized in this function [-Wmaybe-uninitialized]25956 | __pyx_v_ierr = nc_create_par(__pyx_v_path, (__pyx_v_cmode | __pyx_v_parmode), __pyx_v_mpicomm, __pyx_v_mpiinfo, (&__pyx_v_grpid)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:24839:12: note: ‘__pyx_v_mpiinfo’ was declared here24839 | MPI_Info __pyx_v_mpiinfo; | ^~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:25956:30: warning: ‘__pyx_v_mpicomm’ may be used uninitialized in this function [-Wmaybe-uninitialized]25956 | __pyx_v_ierr = nc_create_par(__pyx_v_path, (__pyx_v_cmode | __pyx_v_parmode), __pyx_v_mpicomm, __pyx_v_mpiinfo, (&__pyx_v_grpid)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:24838:12: note: ‘__pyx_v_mpicomm’ was declared here24838 | MPI_Comm __pyx_v_mpicomm; | ^~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:25956:30: warning: ‘__pyx_v_parmode’ may be used uninitialized in this function [-Wmaybe-uninitialized]25956 | __pyx_v_ierr = nc_create_par(__pyx_v_path, (__pyx_v_cmode | __pyx_v_parmode), __pyx_v_mpicomm, __pyx_v_mpiinfo, (&__pyx_v_grpid)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~src/netCDF4/_netCDF4.c:24837:7: note: ‘__pyx_v_parmode’ was declared here24837 | int __pyx_v_parmode; | ^~~~~~~~~~~~~~~
gcc -shared -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/libffi/3.3/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib build/temp.linux-x86_64-cpython-310/src/netCDF4/_netCDF4.o -L/home/user/.local/lib64 -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/lib -L/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.10.2/lib -Wl,--enable-new-dtags,-R/home/user/.local/lib64 -Wl,--enable-new-dtags,-R/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hdf5-mpi/1.12.1/lib -lnetcdf -lhdf5_hl -lhdf5 -o build/lib.linux-x86_64-cpython-310/netCDF4/_netCDF4.cpython-310-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64/eggcreating build/bdist.linux-x86_64/egg/netCDF4creating build/bdist.linux-x86_64/egg/netCDF4/pluginscopying build/lib.linux-x86_64-cpython-310/netCDF4/plugins/empty.txt -> build/bdist.linux-x86_64/egg/netCDF4/pluginscopying build/lib.linux-x86_64-cpython-310/netCDF4/_netCDF4.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/netCDF4copying build/lib.linux-x86_64-cpython-310/netCDF4/_netCDF4.pyx -> build/bdist.linux-x86_64/egg/netCDF4copying build/lib.linux-x86_64-cpython-310/netCDF4/utils.py -> build/bdist.linux-x86_64/egg/netCDF4copying build/lib.linux-x86_64-cpython-310/netCDF4/__init__.py -> build/bdist.linux-x86_64/egg/netCDF4byte-compiling build/bdist.linux-x86_64/egg/netCDF4/utils.py to utils.cpython-310.pycbyte-compiling build/bdist.linux-x86_64/egg/netCDF4/__init__.py to __init__.cpython-310.pyccreating stub loader for netCDF4/_netCDF4.cpython-310-x86_64-linux-gnu.sobyte-compiling build/bdist.linux-x86_64/egg/netCDF4/_netCDF4.py to _netCDF4.cpython-310.pyccreating build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFOcopying src/netCDF4.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFOwriting build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txtzip_safe flag not set; analyzing archive contents...netCDF4.__pycache__.__init__.cpython-310: module references __path__netCDF4.__pycache__._netCDF4.cpython-310: module references __file__creating 'dist/netCDF4-1.7.0-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to itremoving 'build/bdist.linux-x86_64/egg' (and everything under it)Processing netCDF4-1.7.0-py3.10-linux-x86_64.eggremoving '/home/user/virtual-envs/test2/lib/python3.10/site-packages/netCDF4-1.7.0-py3.10-linux-x86_64.egg' (and everything under it)creating /home/user/virtual-envs/test2/lib/python3.10/site-packages/netCDF4-1.7.0-py3.10-linux-x86_64.eggExtracting netCDF4-1.7.0-py3.10-linux-x86_64.egg to /home/user/virtual-envs/test2/lib/python3.10/site-packagesAdding netCDF4 1.7.0 to easy-install.pth fileInstalling nc3tonc4 script to /home/user/virtual-envs/test2/binInstalling nc4tonc3 script to /home/user/virtual-envs/test2/binInstalling ncinfo script to /home/user/virtual-envs/test2/binInstalled /home/user/virtual-envs/test2/lib/python3.10/site-packages/netCDF4-1.7.0-py3.10-linux-x86_64.eggProcessing dependencies for netCDF4==1.7.0Searching for numpy==1.25.2+computecanadaBest match: numpy 1.25.2+computecanadaAdding numpy 1.25.2+computecanada to easy-install.pth fileInstalling f2py script to /home/user/virtual-envs/test2/binInstalling f2py3 script to /home/user/virtual-envs/test2/binInstalling f2py3.10 script to /home/user/virtual-envs/test2/binUsing /home/user/virtual-envs/test2/lib/python3.10/site-packagesSearching for certifi==2023.11.17Best match: certifi 2023.11.17Processing certifi-2023.11.17-py3.10.eggAdding certifi 2023.11.17 to easy-install.pth fileUsing /home/user/virtual-envs/test2/lib/python3.10/site-packages/certifi-2023.11.17-py3.10.eggSearching for cftime==1.6.3Best match: cftime 1.6.3Processing cftime-1.6.3-py3.10-linux-x86_64.eggAdding cftime 1.6.3 to easy-install.pth fileUsing /home/user/virtual-envs/test2/lib/python3.10/site-packages/cftime-1.6.3-py3.10-linux-x86_64.eggFinished processing dependencies for netCDF4==1.7.0
Wondering if anybody has dealt with this issue?
The text was updated successfully, but these errors were encountered:
Installing
netCDF4
(GitHub version with commit f9dc8b2) inside a virtual environment usingpip install
causes an error complaining about the lack ofmpi4py
package. This error is thrown at the user regardless whether thempi4py
is installed within the same virtual environment or not. This is not limited to this version ofnetcdf4-python
, as a few latest releases that I have tried, also demonstrate the same behaviour. Here is an example:However, if the installation proceeds with using the
setup.py
, it works smoothly and there won't be any problem:Wondering if anybody has dealt with this issue?
The text was updated successfully, but these errors were encountered: