Unable to run in Colab or Lightning or GCP VMs - various dependency issues (tried many workarounds)

[devinbost]

I'm running into versioning/dependency issues when trying to get this running.
When I follow the instructions on the readme, I get:

ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from hitter) (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1)
ERROR: No matching distribution found for torch==1.4.0

When I pick one manually to to get beyond the error, I get blocked by a conflict between PyTorch Lightning 1.3.1 and PyYAML.
Both Lightning Studio and Colab use PyYAML 6.0.1, which was released in 2021, by the version of PyTorch Lightning is not compatible with this version.
I have tried manually removing the dependencies and installing from scratch in both environments, but then I get into even deeper dependency conflicts with cython that look like this:

⚡ main ~/HittER pip install path pytorch-pretrained-bert==0.6.0 pytorch_lightning==1.3.1 numba==0.48.0 networkx==2.4
Collecting path
  Using cached path-16.10.0-py3-none-any.whl.metadata (6.4 kB)
Collecting pytorch-pretrained-bert==0.6.0
  Using cached pytorch_pretrained_bert-0.6.0-py3-none-any.whl.metadata (74 kB)
Collecting pytorch_lightning==1.3.1
  Using cached pytorch_lightning-1.3.1-py3-none-any.whl.metadata (24 kB)
Collecting numba==0.48.0
  Using cached numba-0.48.0.tar.gz (2.0 MB)
  Preparing metadata (setup.py) ... done
Collecting networkx==2.4
  Using cached networkx-2.4-py3-none-any.whl.metadata (4.9 kB)
Requirement already satisfied: torch>=0.4.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pytorch-pretrained-bert==0.6.0) (2.2.1)
Requirement already satisfied: numpy in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pytorch-pretrained-bert==0.6.0) (1.26.2)
Requirement already satisfied: boto3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pytorch-pretrained-bert==0.6.0) (1.34.61)
Requirement already satisfied: requests in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pytorch-pretrained-bert==0.6.0) (2.31.0)
Requirement already satisfied: tqdm in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pytorch-pretrained-bert==0.6.0) (4.66.2)
Collecting future>=0.17.1 (from pytorch_lightning==1.3.1)
  Using cached future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
Collecting PyYAML<=5.4.1,>=5.1 (from pytorch_lightning==1.3.1)
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-qhpk1351/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

What's the best path forward here?

[bizhen46766]

Hi, this likely indicates a version conflict. You could try using a slightly newer version of torch, such as 1.11.0 or 1.12.0, to see if it resolves the issue. Since HittER and Relphormer were both open-sourced in 2021, there's a possibility that many libraries do not match up with the latest ones available now.

[devinbost]

Are there requirements to use a specific version of CUDA?

[devinbost]

I tried using this image from GCP:

but after installing conda and running these steps, I get the following error:

conda create -n hitter python=3.7 -y
conda activate hitter
git clone https://github.com/microsoft/HittER.git
cd HittER
pip install -e .

ERROR

Obtaining file:///home/devin_bost/HittER
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of hitter to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from hitter) (from versions: 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1)
ERROR: No matching distribution found for torch==1.4.0

Seems like it's expecting a torch 1.4.0?
If I try to install via:
pip install -r requirements.txt
then I get:

Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: | warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides requested torch >=1.3.1
  - nothing provides requested pytorch_lightning 1.3.1
  - nothing provides requested pytorch-pretrained-bert 0.6.0
  - nothing provides requested torchviz
  - package argparse-1.4.0-py26_0 requires python 2.6*, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ argparse is installable and it requires
│  └─ python 2.6* , which can be installed;
├─ pin-1 is not installable because it requires
│  └─ python 3.7.* , which conflicts with any installable versions previously reported;
├─ pytorch-pretrained-bert 0.6.0  does not exist (perhaps a typo or a missing channel);
├─ pytorch_lightning 1.3.1  does not exist (perhaps a typo or a missing channel);
├─ torch >=1.3.1  does not exist (perhaps a typo or a missing channel);
└─ torchviz does not exist (perhaps a typo or a missing channel).

I'll try PyTorch 2.0 with CUDA 11.8.

[devinbost]

Same behavior.

[devinbost]

When I upgrade setup.py to PyTorch 2.0.0, which requires me to use Python 3.8 (min supported Python version for PyTorch 2), I get this cython error that appears again to relate to PyYAML when I run pip install -e .:

Obtaining file:///home/devin_bost/HittER
  Preparing metadata (setup.py) ... done
Collecting torch==2.0.0 (from HittER==0.1)
  Downloading torch-2.0.0-cp311-cp311-manylinux1_x86_64.whl.metadata (24 kB)
Requirement already satisfied: pyyaml in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from HittER==0.1) (6.0.1)
Requirement already satisfied: pandas in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from HittER==0.1) (2.1.4)
Collecting argparse (from HittER==0.1)
  Downloading argparse-1.4.0-py2.py3-none-any.whl.metadata (2.8 kB)
Collecting path (from HittER==0.1)
  Downloading path-16.10.0-py3-none-any.whl.metadata (6.4 kB)
Requirement already satisfied: sqlalchemy in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from HittER==0.1) (2.0.25)
Collecting torchviz (from HittER==0.1)
  Downloading torchviz-0.0.2.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
Collecting dataclasses (from HittER==0.1)
  Downloading dataclasses-0.6-py3-none-any.whl.metadata (3.0 kB)
Collecting numba==0.48.0 (from HittER==0.1)
  Downloading numba-0.48.0.tar.gz (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pytorch-pretrained-bert==0.6.0 (from HittER==0.1)
  Downloading pytorch_pretrained_bert-0.6.0-py3-none-any.whl.metadata (74 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.7/74.7 kB 14.3 MB/s eta 0:00:00
Collecting networkx==2.4 (from HittER==0.1)
  Downloading networkx-2.4-py3-none-any.whl.metadata (4.9 kB)
Collecting regex==2020.5.14 (from HittER==0.1)
  Downloading regex-2020.5.14.tar.gz (696 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 696.9/696.9 kB 70.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pytorch_lightning==1.3.1 (from HittER==0.1)
  Downloading pytorch_lightning-1.3.1-py3-none-any.whl.metadata (24 kB)
Requirement already satisfied: decorator>=4.3.0 in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from networkx==2.4->HittER==0.1) (5.1.1)
Collecting llvmlite<0.32.0,>=0.31.0dev0 (from numba==0.48.0->HittER==0.1)
  Downloading llvmlite-0.31.0.tar.gz (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.8/110.8 kB 21.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.15 in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from numba==0.48.0->HittER==0.1) (1.26.4)
Requirement already satisfied: setuptools in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from numba==0.48.0->HittER==0.1) (68.2.2)
Requirement already satisfied: future>=0.17.1 in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from pytorch_lightning==1.3.1->HittER==0.1) (0.18.3)
Requirement already satisfied: tqdm>=4.41.0 in /home/devin_bost/anaconda3/lib/python3.11/site-packages (from pytorch_lightning==1.3.1->HittER==0.1) (4.65.0)
Collecting pyyaml (from HittER==0.1)
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 32.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/devin_bost/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/devin_bost/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/devin_bost/anaconda3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/var/tmp/pip-build-env-hawsxex2/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.`

[readytogoyo]

I met the same problem, I changed cuda11.1 and torch 1.8.1, it works.