diff --git a/03_building_and_packaging/intro_slides.md b/03_building_and_packaging/intro_slides.md index 480af3b1..167b91d4 100644 --- a/03_building_and_packaging/intro_slides.md +++ b/03_building_and_packaging/intro_slides.md @@ -30,7 +30,7 @@ slideOptions: ## Learning goals of chapter - Explain why software is packaged. -- Create Python packages, publish on PyPI, and install with pip. +- Create a distributable package of a Python code, publish on PyPI, and install with pip. - Understand the difference between static and dynamic libraries and common ways of installation on Linux. - Build C++ software and handle dependencies with Make and CMake. - Package C++ software with CPack and create Debian packages. @@ -42,17 +42,17 @@ slideOptions: - Bare code is often hard to understand for everyone except the developer(s). - Packaging is a workflow to convert a code into a standardized distributable software. -- A code can be standardized in various ways. Some examples are - - creating a compact form by following a standardization. - - providing an installation recipe, for example, using CMake / make. - - bundling code into an app or software with some UI. -- We discuss **creating a compact form by following a standardization**. +- A code can be standardized in various ways. For example, by ... + - ... providing an installation recipe, for example, using CMake / make. + - ... bundling it into an app or software with a user interface. + - ... packaging it according to an existing standard. +- We discuss **packaging a code according to an existing standard**. --- ## Why should we package code? 1/2 -- A bare code with many files typically has difficulties like +- A code with many files typically has difficulties like - multiple dependencies and requirements of specific versions of dependencies. - intricate compilation / installation steps which are hard to get right. - missing or limited starting information / documentation, which means a high entry barrier. @@ -62,7 +62,7 @@ slideOptions: ## Why should we package code? 2/2 - Create a package to - - benefit from a package index or package manager which is familiar for a broad audience. + - benefit from a package index or package manager which is familiar to a broad audience. - benefit from automated handling of dependencies of package managers. - have ease of distribution and maintenance due to standardization. - increase overall usability and sustainability of your code. @@ -73,20 +73,21 @@ slideOptions: - First step is finding the right standard for your code. - There are several options: - - One of the many Linux package managers: apt, dpkg, yum, RPM and many more ... - - [CMake](https://cmake.org/) : building / installation / packaging tool mostly for C, C++ projects - - [Spack](https://spack.io/) : a package management tool mostly for supercomputing centers - - [Conan](https://conan.io/) : open-source package manager for C and C++ development - - [PyPI](https://pypi.org/) and [pip](https://pypi.org/project/pip/) + - Linux package managers: apt, dpkg, yum, RPM, etc. + - [CMake](https://cmake.org/) + - [Spack](https://spack.io/) + - [Conan](https://conan.io/) + - [pip](https://pypi.org/project/pip/) - [Conda](https://docs.conda.io/en/latest/) + - and many more ... --- ## Why do we look at packaging a Python code? - Python is easy to understand and widely used in research software. -- A well established packaging workflow already exists in the Python community. -- Various examples of packaged codes already exist: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/) and more ... +- Well established package managers and packaging tools already exist in the Python community. +- Several examples of packaged codes: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/). --- @@ -95,4 +96,4 @@ slideOptions: - Packaging or creating build recipe of a code is a standardized process. - Many options in packaging / building tools. - Most of these tools / methods are customized for use cases. -- In this lecture we will concentrate on packaging of Python code. +- In this lecture, we concentrate on packaging of Python code. diff --git a/03_building_and_packaging/pip_demo.md b/03_building_and_packaging/pip_demo.md index 929194b9..a01c7c3a 100644 --- a/03_building_and_packaging/pip_demo.md +++ b/03_building_and_packaging/pip_demo.md @@ -73,11 +73,3 @@ pip show nutils ```bash python -m pip install package-name ``` - -## 4. How to read a PEP - -- Have a look at [PEP 8](https://peps.python.org/pep-0008/) - -## 5. Understanding a PyPI package webpage - -- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index ae50bc76..4a64650f 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -31,431 +31,342 @@ slideOptions: --- -## Python packaging standard is continuously evolving +## Python Package Managers 1/3 -Commonly seen files: - -`setup.py`, `setup.cfg`, `pyproject.toml`, `requirements.txt`. - -All are files which packaging-related tools consume. What do these files do? +Many package managers out there: **pip**, **Conda**, **Poetry**, **uv**, and more. --- -## Python Enhancement Proposals (PEPs) +## Python Package Managers 2/3 -- PEP is an evolving design document which provides information regarding new features of Python, new processes and new environments. -- PEPs typically involve concise technical information, which also acts as standardizations. -- Packaging workflows are also standardized through PEPs. Examples are - - [PEP 427](https://www.python.org/dev/peps/pep-0427/) which introduces the built-package format "wheel". - - [PEP 518](https://peps.python.org/pep-0518/) which introduces a configuration file for packaging. -- Tip: read the *Rationale* section of the **PEP** convention of a particular feature. +pip and Conda are well-established package managers with large ecosystems. ---- +- pip is well-documented and widely used. +- Conda has cross-language support and higher use in scientific computing. -## Python libraries used to install packages +Poetry and uv are newer variants with better functionality and speed. -- `disutils`: old and deprecated ([PEP 632](https://peps.python.org/pep-0632/)). -- `setuptools`: actively maintained packaging tool which is shipped with Python (built on top of `disutils`). -- Many more options, but `setuptools` is lowest common denominator. +- Poetry provides not just packaging but also dependency handling and virtual environment management. +- uv is a faster and smarter drop-in replacement for pip. --- -## setup.py - setup.cfg - pyproject.toml +## Python Package Managers 3/3 + +In this lecture we work with pip, because ... -- Names of all these files are standardized. -- `setup.py` is a script which uses `setuptools`. Needs to be at the root of the repository. -- `setup.cfg` is a config file which has metadata of all the options that can also be specified in `setup.py`. -- `pyproject.toml` has logic and metadata necessary to build and package. +- ... of its wide use in the Python community. +- ... the packaging index PyPI is large. +- ... it is still the go-to solution to package Python code. +- ... it is easy to understand and use. --- -## Comparison of various approaches +## Python Enhancement Proposals (PEPs) -- `setup.py` has been widely popular but main limitation is that it cannot be executed without knowing its dependencies. *Chicken and egg* problem regarding dependencies. -- Does `setup.cfg` solve the dependencies problem? No, because no packaging tool can directly read dependencies from it. -- Solution is to use an additional `pyproject.toml` with the `[build-system]` table specified. -- [PyPA sample project](https://github.com/pypa/sampleproject) shows an example using all three files. +- PEP is an evolving design document which guides Python development. +- PEPs typically involve concise technical information, which also act as standards. +- Packaging is standardized by [Packaging PEPs](https://peps.python.org/topic/packaging/). +- Example of a Packaging PEP: [PEP 427 – The Wheel Binary Package Format 1.0](https://peps.python.org/pep-0427/). --- -## Using only setup.py - -`setup.py` is written using [setuptools](https://pypi.org/project/setuptools/): - -```python -from setuptools import setup -import setuptools - -setup( - name="package-name", - version="", - author="Your Name", - description="A small description", - url="package-website-url", - package_dir={"": ""}, - packages=setuptools.find_packages(where=""), - install_requires=[""] - entry_points={ - 'console_scripts': ['package-import-name = '] - } -) -``` - ---- +## Distribution Package vs. Import Package -## Using setup.cfg and setup.py +Distribution package is ... -Entries moved to `setup.cfg` would look like: +- ... something you can install, like `pip install pkg`. +- ... directly acquired from a packaging index. -```python -[metadata] -name="package-name" -version="" -author="Your Name" -url="package-website-url" -description="A small description" +Import package ... -[options] -packages = find: -install_requires = - "" +- ... is a Python module which typically contains submodules. +- ... is used within a file: `import pkg` or `from pkg import xyz`. +- ... is available when its distribution package is installed. -[options.entry_points] -console_scripts = - executable-name = -``` +Distribution package name and import package need not be the same, but usually is. Read more about [distribution package vs. import package](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/#distribution-package-vs-import-package). -A nominal `setup.py` is still required +--- -```python -from setuptools import setup +## Steps in Packaging a Python Code -if __name__ == "__main__": - setup() -``` +1. Get or create a source tree of the code. +2. Write a packaging configuration file, typically `pyproject.toml`. +3. Create build artifacts. +4. Upload the build artifacts to a packaging index. --- -## Using pyproject.toml - -- According to PEP 621, using `pyproject.toml` is the default recommended way of creating packages with `setuptools`. -- Most important table is `[build-system]` which specifies minimum requirements of the package (PEP 518). -- `pyproject.toml` is readable by packaging tools like pip. - -Example `pyproject.toml` can look like - -```python -[build-system] -requires = ["setuptools", "wheel"] - -[project] -name = "package-name" -description = "A small description" -readme = "README.md" -keywords = ["keyword1", "keyword2"] -classifiers = [ - "Programming Language :: Python :: 3" -] -dependencies = [ - "requests", - 'importlib-metadata; python_version<"3.8"', -] -dynamic = [""] +## Create the source tree 1/3 + +Source tree of code: + +```bash +application/ +├── source.py +├── tests/ ``` +`source.py` contains the code. It can be multiple files. + --- -## Packaging tool `build` +## Create the source tree 2/3 + +Once the file structuring is complete, the source tree is: ```bash -python -m build +application/ +├── LICENSE +├── README.md +├── pyproject.toml +├── package_name/ +| ├── __init__.py +| ├── source.py +└── tests/ ``` -`build` uses `setup.py` for building the package, without any dependency management. - -Drawbacks are +- The file `__init__.py` is required to import the directory `package_name/` as a package. -- Requires manual downloading of files from the package website. -- Packages cannot be easily shared between projects, so you would have to manually define a path which can be used by different projects to access the package. +--- - -Is there a better way? Yes! Use pip! - +## Create the source tree 3/3 ---- +Four places where naming is relevant: -## What is pip? +- Name of the repository (on GitHub or GitLab). +- Name of the folder which has the source code. +- Name of the distribution package. +- Name of the import package. -- pip is a package installer to install packages from the [Python Package Index PyPI]((https://pypi.org/)). -- pip is itself a [package](https://pypi.org/project/pip/) which is available on PyPI. -- pip is open-source and is developed on [GitHub](https://github.com/pypa/pip). +**All names are independent of each other, but try to have a single name.** --- -## Using pip 1/2 +## Write Configuration File 1/4 -Executing: +- File `pyproject.toml` written in the [TOML](https://github.com/toml-lang/toml) language. +- Minimum requirement is specifying a build tool via `[build-system]`. +- In the past, configuration files would be `setup.py` and `setup.cfg`. -```bash -pip install package-name -``` +Documentation about [writing your pyproject.toml](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml). -leads to pip choosing a distribution file for the package and installing it in the environment. +--- -```bash -python -m pip install package-name -``` +## Write Configuration File 2/4 -is basically the same as: +`pyproject.toml` must contain a `[build-system]` table. -```bash -pip install package-name -``` +- `requires` key states which packages are required to build your package. +- `build-backend` key states which build backend tool is used. For example, `setuptools`. +- A build backend converts the source code into a distribution package. ---- +`[project.scripts]` allows defining executable scripts. -## Using pip 2/2 +Complete list of configuration options in [writing your pyproject.toml](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml). -- pip tracks metadata to allow for easy updating and uninstalling of packages. -- pip is bundled together with Python 3.x, making it even easier to use. -- pip can install a package from a source distribution (`.tar.gz`) or a wheel distribution (`.whl`). +--- -**Important**: Do not use +## Write Configuration File 3/4 -```bash -sudo pip install -``` +Let us look at an actual [pyproject.toml](https://github.com/precice/micro-manager/blob/develop/pyproject.toml). -Various security issues with doing so! Go for a venv or +--- -```bash -pip install --user -``` +## Write Configuration File 4/4 ---- +What happened to the files `setup.py` and `setup.cfg`? -## pip vs. pipx +- `setup.py` is a configuration file for the build backend `setuptools`. +- It still remains a valid configuration file. +- Using `setup.py` as a command line tool is **deprecated**. +- `setup.cfg` is yet another valid configuration file for `setuptools`. +- These files are necessary if the Python package has C extensions. -- pip installs packages in the global namespace. -- pipx installs packages in individual virtual environments. -- pipx is meant to be used for applications run directly via the command line. -- For libraries, the recommended way is using pip in a virtual environment. +Read more in [is setup.py deprecated?](https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#setup-py-deprecated). --- -## Using pip +## Create Build Artifacts 1/3 -Uninstall a package +Source distribution (sdist) vs. built distribution (wheel). -```bash -pip uninstall -``` - -Update a package +Base command ```bash -pip install --upgrade +python3 -m build ``` --- -## What is PyPI? - -- [PyPI](https://pypi.org/) = **Python Package Index** is a repository of software developed in the Python community. -- PyPI itself is developed on GitHub through another software called [Warehouse](https://github.com/pypa/warehouse). -- PyPI has an informative [public dashboard](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd) to show its activity. -- A major advantage is the active maintenance of PyPI and the packages indexed in it. -- Not to be confused with **PyPA** which is Python Packaging Authority, a working group which maintains projects in Python packaging. +## Create Build Artifacts 2/3 ---- +Source distribution (sdist) -## Example of PyPI package: fenicsprecice +- Contains files enough to install the package from source. +- Run -- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) + ```bash + python3 -m build --sdist source-tree-directory + ``` --- -## File structure for code packaging 1/2 +## Create Build Artifacts 3/3 -Four places where naming is relevant: +Built distribution (wheel) -- Name of the repository (on GitHub or GitLab). -- Name of the folder which has the source code. -- Name of the package as seen by PyPI. -- Name of the package to be used in the `import` statement. +- Contains files needed only to run package. +- No compilation done, just a copy paste into a directory. +- In most cases only one generic wheel is required. Exceptions are different Python interpreters, different OS configurations. +- Run -**All names are independent of each other.** + ```bash + python3 -m build --wheel source-tree-directory + ``` -Example folder structure: +- If no wheel is available, pip falls back to source distribution. -```bash -generic_folder_name/ -- src/ - - __init__.py - - source-code.py -``` +--- -- The file `__init__.py` is required to import the `package_name/` as a package. This file is mostly empty -- `source-code.py` contains the code. It can be multiple files. +## Uploading Build Artifacts ---- +- [twine](https://twine.readthedocs.io/en/latest/) is a tool to upload the build artifacts. +- Run -## File structure for code packaging 2/2 + ```bash + twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl + ``` -- Once the file structuring is complete, the repository will look like: +Use `twine` because ... -```bash -generic_folder_name/ -- LICENSE -- setup.py -- README.md -- src/ - - __init__.py - - source-code.py - - tests/ -``` +- ... it uses secure authentication of the user over HTTPS. +- ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. +- ... encourages users to create distribution files to promote testing before releasing. --- -## Additional options in setup.py 1/3 +## pip 1/3 -- [Classifiers](https://pypi.org/classifiers/): additional metadata for the version of the package -- Defined as part of [PEP 303](https://www.python.org/dev/peps/pep-0301/#distutils-trove-classification). -- Example: +- pip is a package installer to install packages from the Python package index [PyPI]((https://pypi.org/)). +- pip is itself a [package](https://pypi.org/project/pip/) which is available on PyPI. +- pip is open-source and is developed on [GitHub](https://github.com/pypa/pip). -```python -from setuptools import setup +--- -setup( - ... - classifiers=[ - "Programming Language :: Python :: ", - "License :: OSI Approved :: ", - "Operating System :: ", - ], - ... -) -``` +## pip Demo --- -## Additional options in setup.py 2/3 +## pip 2/3 -- The file `README.md` can be passed as a long description in the following way: +Executing: -```python -from setuptools import setup +```bash +pip install package-name +``` -with open("README.md", "r", encoding="utf-8") as fh: - long_description = fh.read() +leads to pip choosing a distribution file for the package and installing it in the environment. -setup( - ... - long_description=long_description, - long_description_content_type="text/markdown", - ... -) +```bash +python -m pip install package-name +``` + +is basically the same as: + +```bash +pip install package-name ``` --- -## Additional options in setup.py 3/3 +## pip 3/3 -The option `entry_points` exposes code for direct use. For example: executable functions from a terminal. +- pip tracks metadata to allow for easy updating and uninstalling of packages. +- pip is bundled together with Python 3.x, making it even easier to use. +- pip can install a package from a source distribution (`.tar.gz`) or a wheel distribution (`.whl`). -```python -from setuptools import setup +**Important**: Do not use + +```bash +sudo pip install +``` -with open("README.md", "r", encoding="utf-8") as fh: - long_description = fh.read() +as there are security issues! Use a virtual environment or use -setup( - ... - entry_points={ - 'console_scripts': ['package-import-name = '] - } - ... -) +```bash +pip install --user ``` --- -## README.md +## pip vs. pipx -- A `README.md` file typically gives a description of the package, code structure, meta information, the installation procedure and more ... -- It is common practice to parse the `README.md` to `setuptools` as a long description. -- Newer versions of `setuptools` also include this file automatically. -- This file can also be written using [reStructuredText](https://docutils.sourceforge.io/rst.html) which is part of [Docutils](https://docutils.sourceforge.io/index.html). +- pip installs packages in the global namespace. +- pipx installs packages in individual virtual environments. +- pipx is meant to be used for applications run directly via the command line. +- For libraries, the recommended way is using pip in a virtual environment. --- -## Creating distribution archives 1/2 +## pip Commands 1/2 -There are two ways to create these archives: +Uninstall a package + +```bash +pip uninstall +``` -- Using the package builder [build](https://pypa-build.readthedocs.io/en/stable/index.html) which was introduced in [PEP 517](https://www.python.org/dev/peps/pep-0517/). -- Using the packaging standard [wheel](https://wheel.readthedocs.io/en/stable/) which was introduced in [PEP 427](https://www.python.org/dev/peps/pep-0427/). +Update a package -In both cases the files will be generated in a folder `repository/dist/`. +```bash +pip install --upgrade +``` --- -## Creating distribution archives 2/2 +## pip Commands 2/2 -A source distribution archive file (`package-name-.tar.gz`) can be generated by running the command: +List packages by running ```bash -python setup.py sdist +pip freeze ``` -A wheel archive file (`package-name-.whl`) can be generated by running the command: +also useful to generate `requirements.txt`: ```bash -python setup.py bdist_wheel +pip freeze > requirements.txt ``` -Both the commands must be run in the same directory as `setup.py`. +Additionally what works is + +```bash +pipx list +``` --- -## Uploading the distribution archives 1/3 +## PyPI -- [Twine](https://twine.readthedocs.io/en/latest/) is a common tool to upload our package to PyPI. -- Why Twine? - - Secure authentication of the user to PyPI over HTTPS using a verified connection. - - Its predecessor `python setup.py upload` required careful configuration. - - Encourages users to create distribution files to promote testing before releasing. -- The archive files are uploaded to a package index from where pip can get them. +- [PyPI](https://pypi.org/) = **Python Package Index** is an online index of Python packages. +- PyPI itself is developed on GitHub through another software called [Warehouse](https://github.com/pypa/warehouse). +- PyPI has an informative [public dashboard](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd) to show its activity. +- A major advantage is the active maintenance of PyPI and the packages indexed in it. --- -## Uploading the distribution archives 2/3 +## TestPyPI - [TestPyPI](https://test.pypi.org/) is a testing instance of PyPI which does not affect the real index. - Uploading the distribution to TestPyPI before PyPI is a standard pre-deployment step. - Before using TestPyPI (or PyPI) you need an account and a PyPI API Token. - Account creation and API Token generation is straightforward through the [registration page](https://test.pypi.org/account/register/). -- The API token can be configured in a file `$HOME/.pypirc` +- The API token is configured in a file `.pypirc` that is typically in the user home directory. --- -## Uploading the distribution archives 3/3 - -Uploading to TestPyPI can be done using the command: - -```bash -python -m twine upload --repository testpypi dist/* -``` - -The uploading process looks something like: - -```bash -Uploading distributions to https://test.pypi.org/legacy/ -Enter your username: [your username] -Enter your password: [your password] -Uploading package_name-0.0.1.tar.gz -100%| | 4.25k/4.25k [00:01<00:00, 3.05kB/s] -``` +## Example of PyPI package: [fenicsprecice](https://pypi.org/project/fenicsprecice/) --- @@ -465,32 +376,16 @@ Uploading package_name-0.0.1.tar.gz - The installation can be done by: ```bash -pip install --user --index-url https://test.pypi.org/simple/ +pip install --index-url https://test.pypi.org/simple/ ``` --- -## Conda - -- [Conda](https://docs.conda.io/en/latest/) is a package and environment management system which supports multiple languages. -- Conda provides a fast option to setup an isolated environment on your local system. -- Conda is configured to work with [Anaconda installers and packages](https://repo.anaconda.com/). -- Conda is often the preferred way to run Python packages on Windows and MacOS. See example in [FEniCS on Anaconda section](https://fenicsproject.org/download/). - -## Be careful while using Python environments - - - -[xkcd Python Environments](https://imgs.xkcd.com/comics/python_environment.png) - ---- - ## Important takeaways -Packaging of Python code is - -- creation of a standardized folder structure to convert raw Python code into a project. -- creating and uploading distribution archives. -- Uploading distribution archives to a package index. +Packaging of Python code consists of ... -We saw the process of packaging and uploading to TestPyPI which is similar to uploading to the PyPI. +- ... creating a standardized folder structure to convert raw Python code into a project. +- ... creating build artifacts. +- ... uploading build artifacts to a package index. +- ... testing if the uploaded package can be installed. diff --git a/timetable.md b/timetable.md index ba220d58..64271511 100644 --- a/timetable.md +++ b/timetable.md @@ -45,3 +45,12 @@ ## 4.2 – Wed, November 5, 2025 - The Challenge, step one, presentations + +## 5.1 – Wed, November 11, 2025 + +- **15** min.: [Introduction to Packaging](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/intro_slides.md) +- **75** min.: Packaging a Python Code: [demo](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_slides.md), [slides](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_slides.md) + +## 5.2 – Wed, November 11, 2025 + +- **90** min.: [Exercise on Packaging a Python Code](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_exercise.md)