From 4c953502fd51fa02c55a71c4e08f57d98685d896 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Mon, 10 Nov 2025 16:21:47 +0100 Subject: [PATCH 1/6] Rework Introduction to Packaging slides --- 03_building_and_packaging/intro_slides.md | 25 ++++++++++++----------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/03_building_and_packaging/intro_slides.md b/03_building_and_packaging/intro_slides.md index 480af3b1..6e8ab8d1 100644 --- a/03_building_and_packaging/intro_slides.md +++ b/03_building_and_packaging/intro_slides.md @@ -30,7 +30,7 @@ slideOptions: ## Learning goals of chapter - Explain why software is packaged. -- Create Python packages, publish on PyPI, and install with pip. +- Create a distributable package of a raw Python code, publish on PyPI, and install with pip. - Understand the difference between static and dynamic libraries and common ways of installation on Linux. - Build C++ software and handle dependencies with Make and CMake. - Package C++ software with CPack and create Debian packages. @@ -42,17 +42,17 @@ slideOptions: - Bare code is often hard to understand for everyone except the developer(s). - Packaging is a workflow to convert a code into a standardized distributable software. -- A code can be standardized in various ways. Some examples are - - creating a compact form by following a standardization. - - providing an installation recipe, for example, using CMake / make. - - bundling code into an app or software with some UI. -- We discuss **creating a compact form by following a standardization**. +- A code can be standardized in various ways. For example, by ... + - ... providing an installation recipe, for example, using CMake / make. + - ... bundling it into an app or software with a user interface. + - ... packaging it according to an existing standard. +- We discuss **packaging a code according to an existing standard**. --- ## Why should we package code? 1/2 -- A bare code with many files typically has difficulties like +- A code with many files typically has difficulties like - multiple dependencies and requirements of specific versions of dependencies. - intricate compilation / installation steps which are hard to get right. - missing or limited starting information / documentation, which means a high entry barrier. @@ -62,7 +62,7 @@ slideOptions: ## Why should we package code? 2/2 - Create a package to - - benefit from a package index or package manager which is familiar for a broad audience. + - benefit from a package index or package manager which is familiar to a broad audience. - benefit from automated handling of dependencies of package managers. - have ease of distribution and maintenance due to standardization. - increase overall usability and sustainability of your code. @@ -77,16 +77,17 @@ slideOptions: - [CMake](https://cmake.org/) : building / installation / packaging tool mostly for C, C++ projects - [Spack](https://spack.io/) : a package management tool mostly for supercomputing centers - [Conan](https://conan.io/) : open-source package manager for C and C++ development - - [PyPI](https://pypi.org/) and [pip](https://pypi.org/project/pip/) + - [pip](https://pypi.org/project/pip/) - [Conda](https://docs.conda.io/en/latest/) + - and many more ... --- ## Why do we look at packaging a Python code? - Python is easy to understand and widely used in research software. -- A well established packaging workflow already exists in the Python community. -- Various examples of packaged codes already exist: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/) and more ... +- Well established package managers and packaging tools already exist in the Python community. +- Several examples of packaged codes: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/) and more ... --- @@ -95,4 +96,4 @@ slideOptions: - Packaging or creating build recipe of a code is a standardized process. - Many options in packaging / building tools. - Most of these tools / methods are customized for use cases. -- In this lecture we will concentrate on packaging of Python code. +- In this lecture, we concentrate on packaging of Python code. From e945c622dd067e6527d4426d3eea76ff6782cc80 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Tue, 11 Nov 2025 13:04:24 +0100 Subject: [PATCH 2/6] Restructuring Python packaging lecture and adding an entry to the timetable of this semester --- 03_building_and_packaging/pypi_slides.md | 467 ++++++++--------------- timetable.md | 9 + 2 files changed, 177 insertions(+), 299 deletions(-) diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index ae50bc76..7216ac33 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -31,431 +31,315 @@ slideOptions: --- -## Python packaging standard is continuously evolving +## Python Package Managers 1/3 -Commonly seen files: +The most significant package managers: -`setup.py`, `setup.cfg`, `pyproject.toml`, `requirements.txt`. - -All are files which packaging-related tools consume. What do these files do? +- pip +- Conda +- Poetry +- uv --- -## Python Enhancement Proposals (PEPs) +## Python Package Managers 2/3 -- PEP is an evolving design document which provides information regarding new features of Python, new processes and new environments. -- PEPs typically involve concise technical information, which also acts as standardizations. -- Packaging workflows are also standardized through PEPs. Examples are - - [PEP 427](https://www.python.org/dev/peps/pep-0427/) which introduces the built-package format "wheel". - - [PEP 518](https://peps.python.org/pep-0518/) which introduces a configuration file for packaging. -- Tip: read the *Rationale* section of the **PEP** convention of a particular feature. +pip and Conda are well-established package managers with large ecosystems. ---- +- pip is well-documented and widely used. +- Conda has cross-language support and higher use in scientific computing. -## Python libraries used to install packages +Poetry and uv are newer variants with better functionality and speed. -- `disutils`: old and deprecated ([PEP 632](https://peps.python.org/pep-0632/)). -- `setuptools`: actively maintained packaging tool which is shipped with Python (built on top of `disutils`). -- Many more options, but `setuptools` is lowest common denominator. +- Poetry provides not just packaging but also dependency handling and virtual environment management. +- uv is a faster and smarter drop-in replacement for pip. --- -## setup.py - setup.cfg - pyproject.toml - -- Names of all these files are standardized. -- `setup.py` is a script which uses `setuptools`. Needs to be at the root of the repository. -- `setup.cfg` is a config file which has metadata of all the options that can also be specified in `setup.py`. -- `pyproject.toml` has logic and metadata necessary to build and package. - ---- +## Python Package Managers 3/3 -## Comparison of various approaches +In this lecture, we work with pip, because ... -- `setup.py` has been widely popular but main limitation is that it cannot be executed without knowing its dependencies. *Chicken and egg* problem regarding dependencies. -- Does `setup.cfg` solve the dependencies problem? No, because no packaging tool can directly read dependencies from it. -- Solution is to use an additional `pyproject.toml` with the `[build-system]` table specified. -- [PyPA sample project](https://github.com/pypa/sampleproject) shows an example using all three files. +- ... of its wide use in the Python community. +- ... the package repository PyPI is large. +- ... it is still the fundamental way to package Python code. +- ... it is easy to understand and use. --- -## Using only setup.py - -`setup.py` is written using [setuptools](https://pypi.org/project/setuptools/): - -```python -from setuptools import setup -import setuptools - -setup( - name="package-name", - version="", - author="Your Name", - description="A small description", - url="package-website-url", - package_dir={"": ""}, - packages=setuptools.find_packages(where=""), - install_requires=[""] - entry_points={ - 'console_scripts': ['package-import-name = '] - } -) -``` +## Python Enhancement Proposals (PEPs) + +- PEP is an evolving design document which provides information regarding new features, processes and new environments. +- PEPs typically involve concise technical information, which also acts as standardizations. +- Packaging is standardized by [Packaging PEPs](https://peps.python.org/topic/packaging/). +- Example of a Packaging PEP: --- -## Using setup.cfg and setup.py +## Distribution Package vs. Import Package -Entries moved to `setup.cfg` would look like: +Distribution package -```python -[metadata] -name="package-name" -version="" -author="Your Name" -url="package-website-url" -description="A small description" +- Something you can install. +- `pip install pkg`. -[options] -packages = find: -install_requires = - "" +Import package -[options.entry_points] -console_scripts = - executable-name = -``` +- An import package is a Python module which typically contains submodules. +- Writing `import pkg` in a file. Potentially also `from pkg import xyz`. +- Import package is available when its distribution package is installed. -A nominal `setup.py` is still required +Distribution package name and import package need not be the same, but usually is. -```python -from setuptools import setup - -if __name__ == "__main__": - setup() -``` +Read more about [distribution package vs. import package](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/#distribution-package-vs-import-package). --- -## Using pyproject.toml - -- According to PEP 621, using `pyproject.toml` is the default recommended way of creating packages with `setuptools`. -- Most important table is `[build-system]` which specifies minimum requirements of the package (PEP 518). -- `pyproject.toml` is readable by packaging tools like pip. - -Example `pyproject.toml` can look like - -```python -[build-system] -requires = ["setuptools", "wheel"] - -[project] -name = "package-name" -description = "A small description" -readme = "README.md" -keywords = ["keyword1", "keyword2"] -classifiers = [ - "Programming Language :: Python :: 3" -] -dependencies = [ - "requests", - 'importlib-metadata; python_version<"3.8"', -] -dynamic = [""] -``` - ---- +## Steps in Packaging a Python Code -## Packaging tool `build` +1. Get or create a source tree of the code. +2. Write a packaging configuration file, typically `pyproject.toml`. +3. Create build artifacts. +4. Upload the build artifacts to the package distribution service (PyPI). -```bash -python -m build -``` +--- -`build` uses `setup.py` for building the package, without any dependency management. +## Create the source tree 1/2 -Drawbacks are +Four places where naming is relevant: -- Requires manual downloading of files from the package website. -- Packages cannot be easily shared between projects, so you would have to manually define a path which can be used by different projects to access the package. +- Name of the repository (on GitHub or GitLab). +- Name of the folder which has the source code. +- Name of the package as seen by PyPI. +- Name of the package to be used in the `import` statement. - -Is there a better way? Yes! Use pip! - +**All names are independent of each other.** ---- +Example folder structure: -## What is pip? +```bash +generic_folder_name/ +- src/ + - __init__.py + - source-code.py +``` -- pip is a package installer to install packages from the [Python Package Index PyPI]((https://pypi.org/)). -- pip is itself a [package](https://pypi.org/project/pip/) which is available on PyPI. -- pip is open-source and is developed on [GitHub](https://github.com/pypa/pip). +- The file `__init__.py` is required to import the `package_name/` as a package. This file is mostly empty +- `source-code.py` contains the code. It can be multiple files. --- -## Using pip 1/2 +## Create the source tree 2/2 -Executing: +- Once the file structuring is complete, the source tree is: ```bash -pip install package-name +generic_folder_name/ +- pyproject.toml +- src/ + - __init__.py + - source-code.py + - tests/ ``` -leads to pip choosing a distribution file for the package and installing it in the environment. +--- -```bash -python -m pip install package-name -``` +## Write Configuration File 1/4 -is basically the same as: +- File `pyproject.toml` written in the [TOML](https://github.com/toml-lang/toml) language. +- Minimum requirement is specifying a build tool via `[build-system]`. +- In the past, configuration files would be `setup.py` and `setup.cfg`. -```bash -pip install package-name -``` +Documentation about [writing your pyproject.toml](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml). --- -## Using pip 2/2 - -- pip tracks metadata to allow for easy updating and uninstalling of packages. -- pip is bundled together with Python 3.x, making it even easier to use. -- pip can install a package from a source distribution (`.tar.gz`) or a wheel distribution (`.whl`). +## Write Configuration File 2/4 -**Important**: Do not use +`pyproject.toml` must contain a `[build-system]` table. -```bash -sudo pip install -``` +- `requires` key states which packages are required to build your package. +- `build-backend` key states which build backend tool is used. For example, `setuptools`. +- A build backend converts the source code into a distribution package. -Various security issues with doing so! Go for a venv or +`[project.scripts]` allows defining executable scripts. -```bash -pip install --user -``` +Complete list of configuration options in [writing your pyproject.toml](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml). --- -## pip vs. pipx +## Write Configuration File 3/4 -- pip installs packages in the global namespace. -- pipx installs packages in individual virtual environments. -- pipx is meant to be used for applications run directly via the command line. -- For libraries, the recommended way is using pip in a virtual environment. +Let us look at an actual [pyproject.toml](https://github.com/precice/micro-manager/blob/develop/pyproject.toml). --- -## Using pip +## Write Configuration File 4/4 -Uninstall a package +What happened to the files `setup.py` and `setup.cfg`? -```bash -pip uninstall -``` - -Update a package +- `setup.py` is a configuration file for the build backend `setuptools`. +- It still remains a valid configuration file. +- Using `setup.py` as a command line tool is **deprecated**, avoid `python setup.py`. +- `setup.cfg` is yet another valid configuration file for `setuptools`. +- These files are necessary if the Python package as C extensions. -```bash -pip install --upgrade -``` +Read more in [is setup.py deprecated?](https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#setup-py-deprecated). --- -## What is PyPI? +## Create Build Artifacts 1/3 -- [PyPI](https://pypi.org/) = **Python Package Index** is a repository of software developed in the Python community. -- PyPI itself is developed on GitHub through another software called [Warehouse](https://github.com/pypa/warehouse). -- PyPI has an informative [public dashboard](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd) to show its activity. -- A major advantage is the active maintenance of PyPI and the packages indexed in it. -- Not to be confused with **PyPA** which is Python Packaging Authority, a working group which maintains projects in Python packaging. +Source distribution (sdist) vs. built distribution (wheel) --- -## Example of PyPI package: fenicsprecice +## Create Build Artifacts 2/3 -- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) +Source distribution (sdist) + +- Contains files enough to install the package from source at end location. +- Run `python3 -m build --sdist source-tree-directory`. --- -## File structure for code packaging 1/2 +## Create Build Artifacts 3/3 -Four places where naming is relevant: - -- Name of the repository (on GitHub or GitLab). -- Name of the folder which has the source code. -- Name of the package as seen by PyPI. -- Name of the package to be used in the `import` statement. +Built distribution (wheel) -**All names are independent of each other.** +- Contains files needed only to run package at end location. +- No compilation done, just a copy paste into a directory at end location. +- In most cases only one generic wheel is required. Exceptions are different Python interpreters, different OS configurations. +- Run `python3 -m build --wheel source-tree-directory`. +- If no wheel is available, pip falls back to source distribution. -Example folder structure: +--- -```bash -generic_folder_name/ -- src/ - - __init__.py - - source-code.py -``` +## Uploading Build Artifacts -- The file `__init__.py` is required to import the `package_name/` as a package. This file is mostly empty -- `source-code.py` contains the code. It can be multiple files. +- The [twine](https://twine.readthedocs.io/en/latest/) is a tool to upload the build artifacts. +- Run `twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl`. +- Why Twine? + - Secure authentication of the user over HTTPS using a verified connection. + - Its predecessor `python setup.py upload` required careful configuration, and is deprecated. + - Encourages users to create distribution files to promote testing before releasing. +- The archive files are uploaded to a package index from where pip can get them. --- -## File structure for code packaging 2/2 - -- Once the file structuring is complete, the repository will look like: +## pip 1/3 -```bash -generic_folder_name/ -- LICENSE -- setup.py -- README.md -- src/ - - __init__.py - - source-code.py - - tests/ -``` +- pip is a package installer to install packages from the Python package index [PyPI]((https://pypi.org/)). +- pip is itself a [package](https://pypi.org/project/pip/) which is available on PyPI. +- pip is open-source and is developed on [GitHub](https://github.com/pypa/pip). --- -## Additional options in setup.py 1/3 - -- [Classifiers](https://pypi.org/classifiers/): additional metadata for the version of the package -- Defined as part of [PEP 303](https://www.python.org/dev/peps/pep-0301/#distutils-trove-classification). -- Example: +## pip 2/3 -```python -from setuptools import setup +Executing: -setup( - ... - classifiers=[ - "Programming Language :: Python :: ", - "License :: OSI Approved :: ", - "Operating System :: ", - ], - ... -) +```bash +pip install package-name ``` ---- - -## Additional options in setup.py 2/3 - -- The file `README.md` can be passed as a long description in the following way: +leads to pip choosing a distribution file for the package and installing it in the environment. -```python -from setuptools import setup +```bash +python -m pip install package-name +``` -with open("README.md", "r", encoding="utf-8") as fh: - long_description = fh.read() +is basically the same as: -setup( - ... - long_description=long_description, - long_description_content_type="text/markdown", - ... -) +```bash +pip install package-name ``` --- -## Additional options in setup.py 3/3 +## pip 3/3 -The option `entry_points` exposes code for direct use. For example: executable functions from a terminal. +- pip tracks metadata to allow for easy updating and uninstalling of packages. +- pip is bundled together with Python 3.x, making it even easier to use. +- pip can install a package from a source distribution (`.tar.gz`) or a wheel distribution (`.whl`). -```python -from setuptools import setup +**Important**: Do not use -with open("README.md", "r", encoding="utf-8") as fh: - long_description = fh.read() +```bash +sudo pip install +``` -setup( - ... - entry_points={ - 'console_scripts': ['package-import-name = '] - } - ... -) +as there are security issues! Use a virtual environment or use + +```bash +pip install --user ``` --- -## README.md +## pip vs. pipx -- A `README.md` file typically gives a description of the package, code structure, meta information, the installation procedure and more ... -- It is common practice to parse the `README.md` to `setuptools` as a long description. -- Newer versions of `setuptools` also include this file automatically. -- This file can also be written using [reStructuredText](https://docutils.sourceforge.io/rst.html) which is part of [Docutils](https://docutils.sourceforge.io/index.html). +- pip installs packages in the global namespace. +- pipx installs packages in individual virtual environments. +- pipx is meant to be used for applications run directly via the command line. +- For libraries, the recommended way is using pip in a virtual environment. --- -## Creating distribution archives 1/2 +## pip Commands 1/2 -There are two ways to create these archives: +Uninstall a package -- Using the package builder [build](https://pypa-build.readthedocs.io/en/stable/index.html) which was introduced in [PEP 517](https://www.python.org/dev/peps/pep-0517/). -- Using the packaging standard [wheel](https://wheel.readthedocs.io/en/stable/) which was introduced in [PEP 427](https://www.python.org/dev/peps/pep-0427/). +```bash +pip uninstall +``` -In both cases the files will be generated in a folder `repository/dist/`. +Update a package + +```bash +pip install --upgrade +``` --- -## Creating distribution archives 2/2 +## pip Commands 2/2 -A source distribution archive file (`package-name-.tar.gz`) can be generated by running the command: +List packages by running ```bash -python setup.py sdist +pip freeze ``` -A wheel archive file (`package-name-.whl`) can be generated by running the command: +or ```bash -python setup.py bdist_wheel +pipx list ``` -Both the commands must be run in the same directory as `setup.py`. - --- -## Uploading the distribution archives 1/3 +## PyPI -- [Twine](https://twine.readthedocs.io/en/latest/) is a common tool to upload our package to PyPI. -- Why Twine? - - Secure authentication of the user to PyPI over HTTPS using a verified connection. - - Its predecessor `python setup.py upload` required careful configuration. - - Encourages users to create distribution files to promote testing before releasing. -- The archive files are uploaded to a package index from where pip can get them. +- [PyPI](https://pypi.org/) = **Python Package Index** is a repository of software developed in the Python community. +- PyPI itself is developed on GitHub through another software called [Warehouse](https://github.com/pypa/warehouse). +- PyPI has an informative [public dashboard](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd) to show its activity. +- A major advantage is the active maintenance of PyPI and the packages indexed in it. --- -## Uploading the distribution archives 2/3 +## TestPyPI - [TestPyPI](https://test.pypi.org/) is a testing instance of PyPI which does not affect the real index. - Uploading the distribution to TestPyPI before PyPI is a standard pre-deployment step. - Before using TestPyPI (or PyPI) you need an account and a PyPI API Token. - Account creation and API Token generation is straightforward through the [registration page](https://test.pypi.org/account/register/). -- The API token can be configured in a file `$HOME/.pypirc` +- The API token is configured in a file `.pypirc` that is typically in the user home directory. --- -## Uploading the distribution archives 3/3 - -Uploading to TestPyPI can be done using the command: - -```bash -python -m twine upload --repository testpypi dist/* -``` - -The uploading process looks something like: +## Example of PyPI package: fenicsprecice -```bash -Uploading distributions to https://test.pypi.org/legacy/ -Enter your username: [your username] -Enter your password: [your password] -Uploading package_name-0.0.1.tar.gz -100%| | 4.25k/4.25k [00:01<00:00, 3.05kB/s] -``` +- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) --- @@ -465,26 +349,11 @@ Uploading package_name-0.0.1.tar.gz - The installation can be done by: ```bash -pip install --user --index-url https://test.pypi.org/simple/ +pip install --index-url https://test.pypi.org/simple/ ``` --- -## Conda - -- [Conda](https://docs.conda.io/en/latest/) is a package and environment management system which supports multiple languages. -- Conda provides a fast option to setup an isolated environment on your local system. -- Conda is configured to work with [Anaconda installers and packages](https://repo.anaconda.com/). -- Conda is often the preferred way to run Python packages on Windows and MacOS. See example in [FEniCS on Anaconda section](https://fenicsproject.org/download/). - -## Be careful while using Python environments - - - -[xkcd Python Environments](https://imgs.xkcd.com/comics/python_environment.png) - ---- - ## Important takeaways Packaging of Python code is diff --git a/timetable.md b/timetable.md index ba220d58..64271511 100644 --- a/timetable.md +++ b/timetable.md @@ -45,3 +45,12 @@ ## 4.2 – Wed, November 5, 2025 - The Challenge, step one, presentations + +## 5.1 – Wed, November 11, 2025 + +- **15** min.: [Introduction to Packaging](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/intro_slides.md) +- **75** min.: Packaging a Python Code: [demo](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_slides.md), [slides](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_slides.md) + +## 5.2 – Wed, November 11, 2025 + +- **90** min.: [Exercise on Packaging a Python Code](https://github.com/Simulation-Software-Engineering/Lecture-Material/blob/main/03_building_and_packaging/pypi_exercise.md) From 8abacb601b4094ef1454453f5ba326f6f99ee721 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Tue, 11 Nov 2025 14:07:54 +0100 Subject: [PATCH 3/6] Further reworking of slides and slimming down the demo --- 03_building_and_packaging/pip_demo.md | 8 --- 03_building_and_packaging/pypi_slides.md | 76 ++++++++++++++---------- 2 files changed, 43 insertions(+), 41 deletions(-) diff --git a/03_building_and_packaging/pip_demo.md b/03_building_and_packaging/pip_demo.md index 929194b9..a01c7c3a 100644 --- a/03_building_and_packaging/pip_demo.md +++ b/03_building_and_packaging/pip_demo.md @@ -73,11 +73,3 @@ pip show nutils ```bash python -m pip install package-name ``` - -## 4. How to read a PEP - -- Have a look at [PEP 8](https://peps.python.org/pep-0008/) - -## 5. Understanding a PyPI package webpage - -- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index 7216ac33..8e4b5f18 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -72,7 +72,7 @@ In this lecture, we work with pip, because ... - PEP is an evolving design document which provides information regarding new features, processes and new environments. - PEPs typically involve concise technical information, which also acts as standardizations. - Packaging is standardized by [Packaging PEPs](https://peps.python.org/topic/packaging/). -- Example of a Packaging PEP: +- Example of a Packaging PEP: [PEP 427 – The Wheel Binary Package Format 1.0](https://peps.python.org/pep-0427/). --- @@ -82,6 +82,7 @@ Distribution package - Something you can install. - `pip install pkg`. +- Directly acquired from a packaging index. Import package @@ -104,43 +105,49 @@ Read more about [distribution package vs. import package](https://packaging.pyth --- -## Create the source tree 1/2 +## Create the source tree 1/3 -Four places where naming is relevant: +Source tree of code: -- Name of the repository (on GitHub or GitLab). -- Name of the folder which has the source code. -- Name of the package as seen by PyPI. -- Name of the package to be used in the `import` statement. +```bash +application/ +├── source.py +├── tests/ +``` -**All names are independent of each other.** +`source.py` contains the code. It can be multiple files. -Example folder structure: +--- + +## Create the source tree 2/3 + +Once the file structuring is complete, the source tree is: ```bash -generic_folder_name/ -- src/ - - __init__.py - - source-code.py +application/ +├── LICENSE +├── README.md +├── pyproject.toml +├── package_name/ +| ├── __init__.py +| ├── source.py +└── tests/ ``` -- The file `__init__.py` is required to import the `package_name/` as a package. This file is mostly empty -- `source-code.py` contains the code. It can be multiple files. +- The file `__init__.py` is required to import the directory `package_name/` as a package. --- -## Create the source tree 2/2 +## Create the source tree 3/3 -- Once the file structuring is complete, the source tree is: +Four places where naming is relevant: -```bash -generic_folder_name/ -- pyproject.toml -- src/ - - __init__.py - - source-code.py - - tests/ -``` +- Name of the repository (on GitHub or GitLab). +- Name of the folder which has the source code. +- Name of the distribution package. +- Name of the import package. + +**All names are independent of each other, but try to have one name.** --- @@ -192,6 +199,8 @@ Read more in [is setup.py deprecated?](https://packaging.python.org/en/latest/di Source distribution (sdist) vs. built distribution (wheel) +Base command is `python3 -m build`. + --- ## Create Build Artifacts 2/3 @@ -310,7 +319,9 @@ List packages by running pip freeze ``` -or +also useful to generate `requirements.txt`: `pip freeze > requirements.txt`. + +Additionally what works is ```bash pipx list @@ -320,7 +331,7 @@ pipx list ## PyPI -- [PyPI](https://pypi.org/) = **Python Package Index** is a repository of software developed in the Python community. +- [PyPI](https://pypi.org/) = **Python Package Index** is an online index of Python packages. - PyPI itself is developed on GitHub through another software called [Warehouse](https://github.com/pypa/warehouse). - PyPI has an informative [public dashboard](https://p.datadoghq.com/sb/7dc8b3250-85dcf667bd) to show its activity. - A major advantage is the active maintenance of PyPI and the packages indexed in it. @@ -356,10 +367,9 @@ pip install --index-url https://test.pypi.org/simple/ ## Important takeaways -Packaging of Python code is - -- creation of a standardized folder structure to convert raw Python code into a project. -- creating and uploading distribution archives. -- Uploading distribution archives to a package index. +Packaging of Python code consists of ... -We saw the process of packaging and uploading to TestPyPI which is similar to uploading to the PyPI. +- ... creating a standardized folder structure to convert raw Python code into a project. +- ... creating build artifacts. +- ... uploading build artifacts to a package index. +- ... testing if the uploaded package can be installed. From fbe8c77176696af866c8370f92bd3d74695fc8a8 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Tue, 11 Nov 2025 15:01:41 +0100 Subject: [PATCH 4/6] Adjusting Python packaging slides according to slide mode on HedgeDoc --- 03_building_and_packaging/intro_slides.md | 12 +-- 03_building_and_packaging/pypi_slides.md | 105 ++++++++++++---------- 2 files changed, 65 insertions(+), 52 deletions(-) diff --git a/03_building_and_packaging/intro_slides.md b/03_building_and_packaging/intro_slides.md index 6e8ab8d1..167b91d4 100644 --- a/03_building_and_packaging/intro_slides.md +++ b/03_building_and_packaging/intro_slides.md @@ -30,7 +30,7 @@ slideOptions: ## Learning goals of chapter - Explain why software is packaged. -- Create a distributable package of a raw Python code, publish on PyPI, and install with pip. +- Create a distributable package of a Python code, publish on PyPI, and install with pip. - Understand the difference between static and dynamic libraries and common ways of installation on Linux. - Build C++ software and handle dependencies with Make and CMake. - Package C++ software with CPack and create Debian packages. @@ -73,10 +73,10 @@ slideOptions: - First step is finding the right standard for your code. - There are several options: - - One of the many Linux package managers: apt, dpkg, yum, RPM and many more ... - - [CMake](https://cmake.org/) : building / installation / packaging tool mostly for C, C++ projects - - [Spack](https://spack.io/) : a package management tool mostly for supercomputing centers - - [Conan](https://conan.io/) : open-source package manager for C and C++ development + - Linux package managers: apt, dpkg, yum, RPM, etc. + - [CMake](https://cmake.org/) + - [Spack](https://spack.io/) + - [Conan](https://conan.io/) - [pip](https://pypi.org/project/pip/) - [Conda](https://docs.conda.io/en/latest/) - and many more ... @@ -87,7 +87,7 @@ slideOptions: - Python is easy to understand and widely used in research software. - Well established package managers and packaging tools already exist in the Python community. -- Several examples of packaged codes: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/) and more ... +- Several examples of packaged codes: [NumPy](https://pypi.org/project/numpy/), [SciPy](https://pypi.org/project/scipy/), [PyTorch](https://pypi.org/project/torch/). --- diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index 8e4b5f18..6ea5f5d2 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -33,12 +33,7 @@ slideOptions: ## Python Package Managers 1/3 -The most significant package managers: - -- pip -- Conda -- Poetry -- uv +Many package managers out there: **pip**, **Conda**, **Poetry**, **uv**, and more. --- @@ -58,19 +53,19 @@ Poetry and uv are newer variants with better functionality and speed. ## Python Package Managers 3/3 -In this lecture, we work with pip, because ... +In this lecture we work with pip, because ... - ... of its wide use in the Python community. -- ... the package repository PyPI is large. -- ... it is still the fundamental way to package Python code. +- ... the packaging index PyPI is large. +- ... it is still the go-to solution to package Python code. - ... it is easy to understand and use. --- ## Python Enhancement Proposals (PEPs) -- PEP is an evolving design document which provides information regarding new features, processes and new environments. -- PEPs typically involve concise technical information, which also acts as standardizations. +- PEP is an evolving design document which guides Python development. +- PEPs typically involve concise technical information, which also act as standards. - Packaging is standardized by [Packaging PEPs](https://peps.python.org/topic/packaging/). - Example of a Packaging PEP: [PEP 427 – The Wheel Binary Package Format 1.0](https://peps.python.org/pep-0427/). @@ -78,21 +73,15 @@ In this lecture, we work with pip, because ... ## Distribution Package vs. Import Package -Distribution package - -- Something you can install. -- `pip install pkg`. -- Directly acquired from a packaging index. - -Import package - -- An import package is a Python module which typically contains submodules. -- Writing `import pkg` in a file. Potentially also `from pkg import xyz`. -- Import package is available when its distribution package is installed. - -Distribution package name and import package need not be the same, but usually is. +- Distribution package is ... + - ... something you can install, like `pip install pkg`. + - ... directly acquired from a packaging index. +- Import package + - ... is a Python module which typically contains submodules. + - ... is used within a file: `import pkg` or `from pkg import xyz`. + - ... is available when its distribution package is installed. -Read more about [distribution package vs. import package](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/#distribution-package-vs-import-package). +Distribution package name and import package need not be the same, but usually is. Read more about [distribution package vs. import package](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/#distribution-package-vs-import-package). --- @@ -101,7 +90,7 @@ Read more about [distribution package vs. import package](https://packaging.pyth 1. Get or create a source tree of the code. 2. Write a packaging configuration file, typically `pyproject.toml`. 3. Create build artifacts. -4. Upload the build artifacts to the package distribution service (PyPI). +4. Upload the build artifacts to a packaging index. --- @@ -147,7 +136,7 @@ Four places where naming is relevant: - Name of the distribution package. - Name of the import package. -**All names are independent of each other, but try to have one name.** +**All names are independent of each other, but try to have a single name.** --- @@ -187,9 +176,9 @@ What happened to the files `setup.py` and `setup.cfg`? - `setup.py` is a configuration file for the build backend `setuptools`. - It still remains a valid configuration file. -- Using `setup.py` as a command line tool is **deprecated**, avoid `python setup.py`. +- Using `setup.py` as a command line tool is **deprecated**. - `setup.cfg` is yet another valid configuration file for `setuptools`. -- These files are necessary if the Python package as C extensions. +- These files are necessary if the Python package has C extensions. Read more in [is setup.py deprecated?](https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#setup-py-deprecated). @@ -197,9 +186,13 @@ Read more in [is setup.py deprecated?](https://packaging.python.org/en/latest/di ## Create Build Artifacts 1/3 -Source distribution (sdist) vs. built distribution (wheel) +Source distribution (sdist) vs. built distribution (wheel). -Base command is `python3 -m build`. +Base command + +```bash +python3 -m build +``` --- @@ -207,8 +200,12 @@ Base command is `python3 -m build`. Source distribution (sdist) -- Contains files enough to install the package from source at end location. -- Run `python3 -m build --sdist source-tree-directory`. +- Contains files enough to install the package from source. +- Run + + ```bash + python3 -m build --sdist source-tree-directory + ``` --- @@ -216,22 +213,32 @@ Source distribution (sdist) Built distribution (wheel) -- Contains files needed only to run package at end location. -- No compilation done, just a copy paste into a directory at end location. +- Contains files needed only to run package. +- No compilation done, just a copy paste into a directory. - In most cases only one generic wheel is required. Exceptions are different Python interpreters, different OS configurations. -- Run `python3 -m build --wheel source-tree-directory`. +- Run + + ```bash + python3 -m build --wheel source-tree-directory + ``` + - If no wheel is available, pip falls back to source distribution. --- ## Uploading Build Artifacts -- The [twine](https://twine.readthedocs.io/en/latest/) is a tool to upload the build artifacts. -- Run `twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl`. -- Why Twine? - - Secure authentication of the user over HTTPS using a verified connection. - - Its predecessor `python setup.py upload` required careful configuration, and is deprecated. - - Encourages users to create distribution files to promote testing before releasing. +- [twine](https://twine.readthedocs.io/en/latest/) is a tool to upload the build artifacts. +- Run + + ```bash + twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl + ``` + +- Use `twine` because ... + - ... it uses secure authentication of the user over HTTPS. + - ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. + - ... encourages users to create distribution files to promote testing before releasing. - The archive files are uploaded to a package index from where pip can get them. --- @@ -244,6 +251,10 @@ Built distribution (wheel) --- +## pip Demo + +--- + ## pip 2/3 Executing: @@ -319,7 +330,11 @@ List packages by running pip freeze ``` -also useful to generate `requirements.txt`: `pip freeze > requirements.txt`. +also useful to generate `requirements.txt`: + +```bash +pip freeze > requirements.txt +``` Additionally what works is @@ -348,9 +363,7 @@ pipx list --- -## Example of PyPI package: fenicsprecice - -- Having a look at [fenicsprecice](https://pypi.org/project/fenicsprecice/) +## Example of PyPI package: [fenicsprecice](https://pypi.org/project/fenicsprecice/) --- From dc3c509abba1e1994e249f5e125ccfade56b7fc1 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Tue, 11 Nov 2025 15:03:33 +0100 Subject: [PATCH 5/6] Formatting --- 03_building_and_packaging/pypi_slides.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index 6ea5f5d2..df4ab4e9 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -217,7 +217,7 @@ Built distribution (wheel) - No compilation done, just a copy paste into a directory. - In most cases only one generic wheel is required. Exceptions are different Python interpreters, different OS configurations. - Run - + ```bash python3 -m build --wheel source-tree-directory ``` @@ -236,9 +236,9 @@ Built distribution (wheel) ``` - Use `twine` because ... - - ... it uses secure authentication of the user over HTTPS. - - ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. - - ... encourages users to create distribution files to promote testing before releasing. + - ... it uses secure authentication of the user over HTTPS. + - ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. + - ... encourages users to create distribution files to promote testing before releasing. - The archive files are uploaded to a package index from where pip can get them. --- From 39273be804bc477a69da037df5bcf8ed314f27b1 Mon Sep 17 00:00:00 2001 From: Ishaan Desai Date: Tue, 11 Nov 2025 15:13:07 +0100 Subject: [PATCH 6/6] Fix formatting --- 03_building_and_packaging/pypi_slides.md | 27 +++++++++++++----------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/03_building_and_packaging/pypi_slides.md b/03_building_and_packaging/pypi_slides.md index df4ab4e9..4a64650f 100644 --- a/03_building_and_packaging/pypi_slides.md +++ b/03_building_and_packaging/pypi_slides.md @@ -73,13 +73,16 @@ In this lecture we work with pip, because ... ## Distribution Package vs. Import Package -- Distribution package is ... - - ... something you can install, like `pip install pkg`. - - ... directly acquired from a packaging index. -- Import package - - ... is a Python module which typically contains submodules. - - ... is used within a file: `import pkg` or `from pkg import xyz`. - - ... is available when its distribution package is installed. +Distribution package is ... + +- ... something you can install, like `pip install pkg`. +- ... directly acquired from a packaging index. + +Import package ... + +- ... is a Python module which typically contains submodules. +- ... is used within a file: `import pkg` or `from pkg import xyz`. +- ... is available when its distribution package is installed. Distribution package name and import package need not be the same, but usually is. Read more about [distribution package vs. import package](https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/#distribution-package-vs-import-package). @@ -235,11 +238,11 @@ Built distribution (wheel) twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl ``` -- Use `twine` because ... - - ... it uses secure authentication of the user over HTTPS. - - ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. - - ... encourages users to create distribution files to promote testing before releasing. -- The archive files are uploaded to a package index from where pip can get them. +Use `twine` because ... + +- ... it uses secure authentication of the user over HTTPS. +- ... its predecessor `python setup.py upload` required careful configuration, and is deprecated. +- ... encourages users to create distribution files to promote testing before releasing. ---