Replacing Setup.py with Pyproject.py #279

bshifaw · 2024-02-22T15:57:45Z

The python community is moving towards pyproject as the standard method of specifying a project metadata.

Starting with PEP 621, the Python community selected pyproject.toml as a standard way of specifying project metadata
link

The Cromshell repo has been using a mix of pyproject.py and setup.py, this is currently causing issue during installation.

integration: packaging backend failed (code=1), with AttributeError: 'NoneType' object has no attribute 'split'
/home/runner/work/cromshell/cromshell/.tox/.pkg/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:75: _MissingDynamic: `description` defined outside of `pyproject.toml` is ignored.
!!

        ********************************************************************************
        The following seems to be defined outside of `pyproject.toml`:

        `description = 'Command Line Interface (CLI) for Cromwell servers'`

        According to the spec (see the link below), however, setuptools CANNOT
        consider this value unless `description` is listed as `dynamic`.

        https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

        To prevent this problem, you can list `description` under `dynamic` or alternatively
        remove the `[project]` table from your file and rely entirely on other means of
        configuration.
        ********************************************************************************

Changes in this repo replaceses setup.py completely with pyproject.toml, fixing the installation errors.

…d recursive packaging from test-requirements.txt and dev-requirements.txt, removed setup* from tox.ini,

…alidate pyproject.toml

SHuang-Broad

Tests seem to be failing.

…ted and 3.8 support will end this year

bshifaw · 2024-02-22T20:00:06Z

.github/workflows/integration_tests.yml

@@ -9,7 +9,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python: [ '3.7', '3.8', '3.9', '3.10' ]


3.7 is no longer being supported and support for 3.8 is ending this year: https://devguide.python.org/versions/

Can you no longer install on python 3.7 /8 now? Or is it just not being tested.

Just out of an abundance of caution we might want to check what version of python the terra and all of us images are using.

Terra/AOU uses python3.10.12
cromshell on p3.7 is no longer being tested

I had some trouble installing dev packages on py3.11 because mypy==0.770 was not supported.
So I updated this to the latest version, the latest version doesn't work on 3.7. None of the packages in the requirements have been updated so cromshell should still work on py3.7 but if anyone is developing on 3.7 they'll run into package issues.

bshifaw · 2024-02-22T20:04:53Z

dev-requirements.txt

 cython
 sphinx
 sphinx-rtd-theme
-mypy==0.770


needed to updated because latest version of Py3.11 did not support this version

bshifaw · 2024-02-22T20:05:10Z

dev-requirements.txt

 tox
 wheel
 bumpversion
 twine
 black
 flake8
 pylint
+tomli==2.0.1


needed for tox validation of pyproject.toml

bshifaw · 2024-02-22T20:06:39Z

dev-requirements.txt

 cython
 sphinx
 sphinx-rtd-theme
-mypy==0.770
-r requirements.txt


pyproject.toml does not support recursive package installation

How do transitive dependencies get installed then?

I mean to say pyproject.toml requires a dependency txt file to list packages directly, and not point to another dependency file.

Transitive dependencies are still handled by pip during installation. The requiremnts.txt file is listed as core dependencies in the pyprojec.toml, and pip will install the transitive dependencies. The dev-requirements and test-requirements are listed as optional dependencies

Thank you for the clarification!

bshifaw · 2024-02-22T20:08:37Z

tox.ini

    pylint --exit-zero src
-
-[gh-actions]


The original thought was to have this be used during github testing, but the github actions list the python version to test so its is not needed here

bshifaw · 2024-02-22T20:10:04Z

.github/workflows/pypi_packaging.yml

@@ -80,9 +80,6 @@ jobs:
    - name: Setup Env
      run: python3 -m pip install --upgrade pip build

-    - name: Validate setup.py


There is no integrated solution for validating the pyproject.toml file. There are some third-party application but they are unstable.
I've noticed though that tox run excutting its test automatically does a basic level check of the toml file using not a cli command but via package called "tomli". This has been added to the dev-requirments.txt file

lbergelson

@bshifaw Looks fine to me, but I definitely don't understand the nuance here.

I think we should probably check with the people using it for the all of us workspaces to make sure they have 3.9 available.

lbergelson · 2024-02-22T20:24:11Z

dev-requirements.txt

 cython
 sphinx
 sphinx-rtd-theme
-mypy==0.770
-r requirements.txt


How do transitive dependencies get installed then?

lbergelson · 2024-02-22T20:24:34Z

pyproject.toml

@@ -1,3 +1,5 @@
+
+ignore_missing_imports = true


Can you explain this one to me? Don't we need all our imports?

Thanks for this catch, its suppose to be under [tool.mypy."pytest.*"]
When ignore_missing_imports is set to true, mypy will not report an error when an imported module cannot be found. This is to replicate the setting that was originally found in the setup.cfg file

lbergelson · 2024-02-22T20:27:12Z

.github/workflows/integration_tests.yml

@@ -9,7 +9,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python: [ '3.7', '3.8', '3.9', '3.10' ]


Can you no longer install on python 3.7 /8 now? Or is it just not being tested.

lbergelson · 2024-02-22T20:30:52Z

.github/workflows/integration_tests.yml

@@ -9,7 +9,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python: [ '3.7', '3.8', '3.9', '3.10' ]


Just out of an abundance of caution we might want to check what version of python the terra and all of us images are using.

bshifaw · 2024-02-22T20:42:01Z

@bshifaw Looks fine to me, but I definitely don't understand the nuance here.

I think we should probably check with the people using it for the all of us workspaces to make sure they have 3.9 available.

This update was to help fix the installation on AOU workspaces. When running their cromshell tutorial, the installation included a cell that updated the cromshell version by pip - git (pip install git+https://github.com/broadinstitute/cromshell). This caused the installation error mentioned in the PR description. Their python version is 3.10.3.

Also, to confirm I just now installed cromshell in an AOU notebook using this specific branch; the instillation now completes successfully.

bshifaw · 2024-02-22T21:33:59Z

tox.ini

@@ -49,9 +53,9 @@ deps =
 		flake8

 commands =
-    black --check --diff --target-version py38 src tests setup.py
-    isort --check-only --profile black --diff -rc tests src setup.py


-rc for isort is deprecated because options now happen by default

bshifaw added 7 commits February 22, 2024 09:04

replaced setup.cfg and setup.py completly with pyproject.toml, remove…

f81c5f4

…d recursive packaging from test-requirements.txt and dev-requirements.txt, removed setup* from tox.ini,

fixed pyproject.toml setuptools package find location

9ffa8e4

fixed dev-requirements.txt updated mypy version

7cd1b04

Removed setup.py check command, currently no integrated solution to v…

ff9cd28

…alidate pyproject.toml

updated dev pip install in README.md

6b7bcea

minor formatting updates

83e711c

minor formatting updates

db65a8b

bshifaw requested a review from SHuang-Broad February 22, 2024 15:57

SHuang-Broad reviewed Feb 22, 2024

View reviewed changes

updated python support to =>3.9 because 3.7 is no longer being suppor…

63d10d3

…ted and 3.8 support will end this year

bshifaw mentioned this pull request Feb 22, 2024

Fix cost (for methods cromwell), add explicit example #278

Merged

bshifaw commented Feb 22, 2024

View reviewed changes

bshifaw added the bug Something isn't working label Feb 22, 2024

lbergelson approved these changes Feb 22, 2024

View reviewed changes

bshifaw added 2 commits February 22, 2024 16:24

moved ignore_missing_imports under pytest in pyproject.toml

dd4feb0

minor formatting change

29d43f6

bshifaw commented Feb 22, 2024

View reviewed changes

bshifaw merged commit 4a485d4 into main Feb 23, 2024
6 checks passed

bshifaw deleted the bs_toml_add_dynamic_key branch February 23, 2024 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replacing Setup.py with Pyproject.py #279

Replacing Setup.py with Pyproject.py #279

bshifaw commented Feb 22, 2024

SHuang-Broad left a comment

bshifaw Feb 22, 2024

lbergelson Feb 22, 2024

lbergelson Feb 22, 2024

bshifaw Feb 22, 2024

bshifaw Feb 22, 2024

bshifaw Feb 22, 2024

bshifaw Feb 22, 2024

lbergelson Feb 22, 2024

bshifaw Feb 22, 2024

lbergelson Feb 23, 2024

bshifaw Feb 22, 2024

bshifaw Feb 22, 2024

lbergelson left a comment

lbergelson Feb 22, 2024

lbergelson Feb 22, 2024

bshifaw Feb 22, 2024

lbergelson Feb 22, 2024

lbergelson Feb 22, 2024

bshifaw commented Feb 22, 2024

bshifaw Feb 22, 2024

Replacing Setup.py with Pyproject.py #279

Replacing Setup.py with Pyproject.py #279

Conversation

bshifaw commented Feb 22, 2024

SHuang-Broad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbergelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bshifaw commented Feb 22, 2024

Choose a reason for hiding this comment