diff --git a/docs/conf.py b/docs/conf.py index d5f786d..0e2f35a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -216,6 +216,7 @@ myst_enable_extensions = { "colon_fence", + "substitution", } # Custom Sphinx extensions; see diff --git a/docs/how-to/cranking/set-up-cranky-environment.md b/docs/how-to/cranking/set-up-cranky-environment.md new file mode 100644 index 0000000..82560fb --- /dev/null +++ b/docs/how-to/cranking/set-up-cranky-environment.md @@ -0,0 +1,69 @@ +# How to set up cranky environment + +This guide describes the one-off process to configure your build environment to +crank Ubuntu kernels using `cranky`. + +## Install dependencies + +First, you will need to install the required packages. + +### Debian packages + +Use apt to install the following debian packages: + +```{code-block} text +sudo apt install \ + bash-completion \ + build-essential \ + ccache \ + debhelper \ + devscripts \ + docbook-utils \ + fakeroot \ + gawk \ + git \ + git-email \ + kernel-wedge \ + libncurses5-dev \ + makedumpfile \ + python3-launchpadlib \ + python3-ruamel.yaml \ + schroot \ + sharutils \ + transfig \ + ubuntu-dev-tools \ + wget \ + xmlto +``` + +### Snapcraft package + +Install snapcraft according to the Ubuntu release running on your build machine. + +```{tip} +You can check your release of Ubuntu by running `lsb_release -a`. +``` + +`````{tab-set} +````{tab-item} 24.04 LTS (Noble Numbat) or later + +Install `snapcraft` using snap: + +```{code-block} text +snap install snapcraft --classic +``` +```` + +````{tab-item} 22.04 LTS (Jammy Jellyfish) or earlier + +Install `snapcraft` using apt: + +```{code-block} text +sudo apt install snapcraft +``` +```` + +````` + + + diff --git a/docs/how-to/index.md b/docs/how-to/index.md index ba070ef..bf5c944 100644 --- a/docs/how-to/index.md +++ b/docs/how-to/index.md @@ -10,6 +10,7 @@ kernel packages and components. Source code access and management Development and customization +Set up cranky environment Test kernels in -proposed Contribute to kernel docs ``` @@ -22,6 +23,14 @@ tree to build custom kernel modules, and more. - [Obtain kernel source for an Ubuntu release using Git](/how-to/source-code/obtain-kernel-source-git) +## Cranking kernels + +In Canonical, "kernel cranking" refers to the process of applying patches and +updates to an Ubuntu kernel, packaging it, and getting it ready for testing -- +all done using the `cranky` toolchain. + +- [Set up cranky environment](/how-to/cranking/set-up-cranky-environment) + ## Development and customization The steps to build a kernel is similar but may have slightly difference diff --git a/docs/index.md b/docs/index.md index c73593b..4f8c8c8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -28,7 +28,7 @@ processes for customization and maintenance. ```{grid-item-card} [Tutorials](/tutorial/index) **Start here**: a hands-on introduction to Ubuntu Linux kernel development for -new users +new users. ``` ```{grid-item-card} [How-to guides](/how-to/index) @@ -73,8 +73,8 @@ fixes and constructive feedback. :hidden: :maxdepth: 2 +/tutorial/index /how-to/index /reference/index /explanation/index -/tutorial/index ``` diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index 8d06c18..52a6f9b 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -47,7 +47,7 @@ linux-meta When a new kernel version is released and marked stable, the linux-meta package is updated to reference the new version, allowing automatic upgrades. -linux-signed +linux-signed Refers to kernel packages that are cryptographically signed to ensure their integrity and authenticity. These signatures are crucial for secure boot environments, as they enable the diff --git a/docs/tutorial/crank-your-first-kernel.md b/docs/tutorial/crank-your-first-kernel.md index 38ae354..fa917bf 100644 --- a/docs/tutorial/crank-your-first-kernel.md +++ b/docs/tutorial/crank-your-first-kernel.md @@ -1,43 +1,50 @@ # Tutorial: Crank your first kernel -## Introduction - -Cranking an Ubuntu kernel is the process of applying updates to an existing Ubuntu kernel, packaging it, and preparing it for testing. [cranky](https://kernel.ubuntu.com/gitea/kernel/kteam-tools/src/branch/master/cranky) is a toolchain which assists in all of these steps. +Cranking an Ubuntu kernel is the process of applying patches and updates to an existing Ubuntu kernel, packaging it, and preparing it for testing. All this done using the [cranky](https://kernel.ubuntu.com/gitea/kernel/kteam-tools/src/branch/master/cranky) toolchain. In this tutorial, we will crank a 24.04 LTS (Noble Numbat) (codename `noble`) Google cloud kernel (codename `linux-gke`). Keep these codenames in mind for future commands. -## 1. Set Up/Update Environment +## Set up and update build environment + + +You will need to complete the setup according to the {doc}`cranky environment setup ` guide before continuing. -First, follow the [Cranky Environment Setup] tutorial. +### Update chroot environment -### 1.1. Update chroot environment -We use [chroot](https://en.wikipedia.org/wiki/Chroot) environments to isolate different sets of tools when kernel cranking. cranky helps us set up and manage these chroot jails. +We use [chroot](https://en.wikipedia.org/wiki/Chroot) environments to isolate different sets of tools when doing kernel cranking. `cranky` helps us set up and manage these chroot jails. -If it's our first time creating a chroot for this release (Noble Numbat), we must first create the chroot base: + +If this is your first time creating a chroot for the Noble Numbat release, you must first create the chroot base: + + ```bash cranky chroot create-base noble:linux-gke ``` -This will output a lot of text as it installs various packages needed for the chroot to work properly. (It took ~20 minutes to complete.) + +This can take up to 20 minutes to complete. +If successful, you should observe the various packages being installed and set up in the terminal output. Next, create the chroot session: + ```bash cranky chroot create-session noble:linux-gke ``` -This will also output a lot of text as it uses apt to install various packages needed for the crank. (It took ~2 minutes to complete.) +This step uses APT to install various packages needed for the crank and takes about two minutes to complete. - +### Update kteam-tools repository -### 1.2. Update kteam-tools -Finally, update your local clone of kteam-tools to the latest commit on the main branch: +Update your local clone of `kteam-tools` to the latest commit on the `master` branch: For the scope of this tutorial, we will use a particular version of cranky. Run the following commands to get it: -:::{WARNING} + + +```{warning} If your kteam-tools tree isn't clean, be sure to save your work before running the commands. They will modify the repository! Git should warn you if any destructive actions might occur. -::: +``` ```bash cd ~/canonical/kteam-tools/ @@ -60,46 +67,60 @@ Note: switching to 'cranky-tutorial'. HEAD is now at de4674e4 mainline-build/cod-update-virgin: Split script and update freedesktop repos ::: -## 2. Download Current Version of the Kernel -Next, clone the chosen kernel in its current state. +## Download current version of kernel + +You're now ready to clone the `linux-gke` kernel in its current state. + + + ```bash cd ~/canonical/kernel/ubuntu/ cranky checkout noble:linux-gke ``` + + Once this command is finished (It took ~20 minutes to complete), you should see the following directories inside the newly-created `./noble/linux-gke/` directory: -- `linux-main/`: The actual Linux kernel source -- `linux-meta/` -- `linux-signed/`: Canonical's kernel signing stuff +- `linux-main`: The actual Linux kernel source. +- `linux-meta`: Stores a set of meta-packages for the kernel. See {term}`linux-meta` for more information. +- `linux-signed`: Kernel packages that are cryptographically signed to ensure their integrity and authenticity. See {term}`linux-signed` for more information. + -## 3. Apply Updates from the Upstream Kernel -The upstream kernel will have changes that should be propagated down to this kernel. +## Apply updates from upstream kernel + +The upstream kernel will possibly have changes that should be propagated down to this kernel. + +### Fix helper scripts -### 3.1. Fix helper scripts Update the local (in-tree) helper scripts cranky uses to the latest version: + ```bash cd ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main/ cranky fix ``` You should see some output showing that cranky executed several scripts. +You see observe cranky going through the updating process for various scripts in the output terminal. +### Rebase on top of updated parent -### 3.2. Rebase on top of updated parent -Because this is a derivative kernel, we need to apply updates from its parent, the generic kernel: +As `linux-gke` is a derivative kernel, we need to apply updates from its parent, the generic kernel: ```bash cranky rebase ``` -:::{tip} -For non-derivative kernels (e.g., `noble:linux`), this step is not required, and doesn't do anything. -::: + + +```{tip} +For non-derivative kernels (e.g., `noble:linux`), this step is not required. +``` + +### Fix helper scripts (again) -### 3.3. Fix helper scripts (again) Sometimes after a `cranky rebase`, the helper scripts get updated. It's good practice to always re-run: ```bash @@ -107,33 +128,91 @@ cranky fix ``` -## 4. TODO new section -### 4.1. Starting commit +## Commit and review updates + +Now that we've pulled in all the upstream changes, we are ready to review and apply the commits to the `linux-gke` kernel. + +### Add starting commit + -Run the following command: +Create a commit that signifies the start of a new release. This new commit will contain {term}`ABI` changes and any customization required by backport kernels. ```bash cranky open ``` -This command creates a commit which signifies the start of a new release. It updates `debian.gke/changelog` to reflect this. -Optionally, it may also update ABI versioning info. +You should observe something similar in the output terminal: - +```{terminal} +:input: cranky chroot create-base noble:linux-gke +:user: kernel-engineer +:host: ubuntu-machine +:dir: ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main +[...] +/home/kernel-engineer/canonical/kteam-tools/cranky/cranky startnewrelease --commit +Creating new changelog set for 6.8.0-1018.22... +[cranky/master-next 7c8a6e1e1f8b] UBUNTU: Start new release + 1 file changed, 8 insertions(+) -Run `git show` and see that this new commit starts with "UBUNTU: Start new release" and shows an update to `./debian.gke/changelog`. +***** Now please inspect the commit before pushing ***** +``` + +Run `git show` to verify that this new commit starts with "UBUNTU: Start new release" and shows an update to {file}`./debian.gke/changelog`. + +```{terminal} +:input: git show +:user: kernel-engineer +:host: ubuntu-machine +:dir: ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main + +Author: kernel-engineer +Date: Mon Jan 20 16:55:52 2025 +0800 + + UBUNTU: Start new release + + Ignore: yes + Signed-off-by: annecyh + +diff --git a/debian.gke/changelog b/debian.gke/changelog +[...] +``` + +### Review rebased changes + +Sometimes the rebase doesn't get all the changes. So we need to run this command to manually review any outstanding changes: -### 4.2. Review -Sometimes the rebase doesn't get all the changes. Run this command to manually review any outstanding changes: ```bash cranky review-master-changes ``` This command will output any outstanding changes in a list with their commit hashes and descriptions. Use `git show ` to view the changes. +```{terminal} +:input: cranky review-master-changes +:user: kernel-engineer +:host: ubuntu-machine +:dir: ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main + +Listing changes in "debian.master/" since 9f8080a647a9e2c8c9a52b3e471b3f22d4dc0c67... + +851f47333771 UBUNTU: [Packaging] debian.master/dkms-versions -- update from kernel-versions (main/2025.01.13) +41b4e628ce36 UBUNTU: Upstream stable to v6.6.55, v6.10.14 +1de0f53d2545 UBUNTU: [Config] updateconfigs for MICROSOFT_MANA +547230d18843 UBUNTU: [Config] updateconfigs for deprecated CONFIG_Z3FOLD +1b64b00b69b7 UBUNTU: [Config] updateconfigs to set ILLEGAL_POINTER_VALUE for riscv64 +14549d19d4b5 UBUNTU: [Config] updateconfigs to select PROC_MEM_ALWAYS_FORCE +3eb67a5e5da8 UBUNTU: [Packaging] Add list of used source files to buildinfo package +3b9e978a6cb4 UBUNTU: [Packaging] Sort build dependencies alphabetically +18d768dbc001 UBUNTU: Upstream stable to v6.6.54, v6.10.13 +0861dae772cb UBUNTU: [Config] update configs for CONFIG_CRYPTO_AES_GCM_P10 +82fbe5ae5484 UBUNTU: Upstream stable to v6.6.53, v6.10.12 +``` + + + Usually these can be ignored, but there are a few instances where further investigation is necessary: - Commits with changes to `debian.master/rules.d/` @@ -142,86 +221,88 @@ Usually these can be ignored, but there are a few instances where further invest - These indicate a change in the parent kernel's configuration. - You'll need to compare this change with what appears in the derivative config (`debian.gke/`) to decide if it should be applied to this crank. -### 4.3. Link to the Launchpad Bug Tracker +### Link to Launchpad bug tracker Run the following command to link this kernel to its corresponding Launchpad bug tracker: + -:::{caution} +```{warning} Use `--dry-run` unless you are actually cranking a kernel. Otherwise, this will overwrite Launchpad and might make destructive changes! -::: +``` ```bash cranky link-tb --dry-run ``` -This updates the Launchpad tracking bug, which, among other things, will be used as an input for subsequent steps. - - - -This will create a git commit. Run `git show` to see it. It should look something like this: +```{tip} +If you are running `cranky link-tb` for the first time, you will be directed to the "Authorize application to access Launchpad on your behalf" page. Choose your preferred option before continuing. +``` -```diff -commit 4345a7fc255b03ff9072cdcec1779a9b39d7519b (HEAD -> cranky/master-next) -Author: Benjamin Wheeler -Date: Wed Jan 15 12:14:59 2025 -0500 +This step should update the Launchpad tracking bug -- which, among other things, will be used as an input for subsequent steps -- and create a git commit. - UBUNTU: link-to-tracker: update tracking bug - - BugLink: https://bugs.launchpad.net/bugs/2093494 - Properties: no-test-build - Signed-off-by: Benjamin Wheeler +But since we used the `--dry-run` option for this tutorial, no changes are made to the local tree and no commit is created. -diff --git a/debian.gke/tracking-bug b/debian.gke/tracking-bug -index 6f5c6b4a700c..b3436ee66bdd 100644 ---- a/debian.gke/tracking-bug -+++ b/debian.gke/tracking-bug -@@ -1 +1 @@ --2090338 s2024.10.28-1 -+2093494 s2024.12.02-1 -``` +```{terminal} +:input: cranky link-tb --dry-run +:user: kernel-engineer +:host: ubuntu-machine +:dir: ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main -Then, click the `BugLink` URL to see the relevant Launchpad tracking bug. -A comment should appear on there that says something like: +(This is a dry-run) +LP: #2093652 (noble/linux-gke: -proposed tracker) 2025.01.13-1 +Dry Run -- no changes made ``` - summary: - noble/linux-gke: -proposed tracker - + noble/linux-gke: 6.8.0-1017.21 -proposed tracker -``` - -If you see similar results, you've successfully linked to the tracking bug. -### 4.4. Update DKMS Packages + + -`debian.master/dkms-versions` specifies dkms modules to be packaged with its kernel. +### Update DKMS packages -This command updates the package versions in `debian.gke/dkms-versions` to match the ones expected for the SRU cycle +The `debian.master/dkms-versions` file specifies dkms modules to be packaged with its kernel. This command updates the package versions in `debian.gke/dkms-versions` to match the ones expected for the SRU cycle. + ```bash cranky update-dkms-versions ``` -No changes are needed this time, so you'll see this output: -``` +Since no changes are needed at this time, you should observe the following output: + +```{terminal} +:input: cranky update-dkms-versions +:user: kernel-engineer +:host: ubuntu-machine +:dir: ~/canonical/kernel/ubuntu/noble/linux-gke/linux-main + +[...] + debian.gke/dkms-versions: No changes from kernel-versions ``` -In most cases there would be no change committed as the up-to-date versions should have been committed on the master kernel and picked-up by the derivative or backport on rebase. However, if there is any change, check that the version numbers only become higher and nothing gets dropped completely. In case anything looks suspicious, don’t hesitate to ask the team if the changes are expected. + +```{tip} +In most cases, no changes are expected as the up-to-date DKMS versions should have been committed on the generic kernel and picked up by the derivative or backport on rebase. +``` + +### Add closing commit +We will now create one final commit before preparing a release by running: -### 4.5. Closing commit -This step creates one final commit before a release is prepared. Run: ```bash cranky close ``` -This command is a shortcut for several steps: +This command is a shortcut that does the following: + + + 1. Verifies there are no changes left. 2. Inserts changes from the parent kernel into the changelog. 3. Inserts git changes into the changelog. 4. Updates the release series, author and date on the changelog, thus closing the changelog. 5. Creates a commit signifying the finished crank. -If the output went well, you should see a new commit when you run `git show`: +If successful, you should see a new commit when you run `git show`: ```diff commit 6c9a5055b22f4c30aa3ba0c9df306762edb29197 (HEAD -> cranky/master-next) Author: Benjamin Wheeler diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md index f609f7d..ab9ded5 100644 --- a/docs/tutorial/index.md +++ b/docs/tutorial/index.md @@ -1,7 +1,7 @@ # Tutorials -The following tutorials will present you with different tutorials that address -key concepts, tools, processes and operations. +The following section will present you with different tutorials that address +key concepts, tools, processes, and operations relevant to the kernel team. ```{toctree}