Repository navigation
Implementing stable package lock for CI builds #1148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f89d731
787e7ae
fd2b47e
59ff923
22f5e1a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| name: Install package manifest | ||
| description: > | ||
| Restore Matter packages. In locked mode, install CI Conan lock derived from | ||
| SLT pkg.lock, then install local matter_app. Also exports CI_PKG_LOCK for | ||
| build_app.sh (slt --check-updates=false). | ||
|
|
||
| inputs: | ||
| package-version: | ||
| description: Version of matter_app to install from the restored Matter cache | ||
| required: true | ||
| conan-cache-key: | ||
| description: Cache key created by the Matter package producer | ||
| required: true | ||
| use-lockfile: | ||
| description: If true, conan install --lockfile + export CI_PKG_LOCK | ||
| required: false | ||
| default: "true" | ||
| fail-on-cache-miss: | ||
| description: Fail when the Matter package cache does not exist | ||
| required: false | ||
| default: "true" | ||
| download-parallel: | ||
| description: Maximum parallel Conan downloads after restore | ||
| required: false | ||
| default: "1" | ||
| lockfile-dir: | ||
| description: Directory with pkg.lock, conan.lock, conanfile.txt | ||
| required: false | ||
| default: .github/pkg-manifest | ||
| prerelease-remote-url: | ||
| description: Conan prerelease remote URL | ||
| required: false | ||
| default: https://conan-prerelease.silabs.net/ | ||
|
|
||
| outputs: | ||
| cache-hit: | ||
| description: Whether the Matter package cache was restored | ||
| value: ${{ steps.restore.outputs.cache-hit }} | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Prepare Conan home | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| : "${CONAN_HOME:?CONAN_HOME must be set (run cache-tools first)}" | ||
| rm -rf "${CONAN_HOME}" | ||
| mkdir -p "${CONAN_HOME}" | ||
|
|
||
| - name: Restore Matter package cache | ||
| id: restore | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: ${{ env.CONAN_HOME }} | ||
| key: ${{ inputs.conan-cache-key }} | ||
| fail-on-cache-miss: ${{ inputs.fail-on-cache-miss }} | ||
|
|
||
| - name: Configure Conan | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| mkdir -p conan-settings | ||
| printf 'core.download:parallel=%s\n' "${{ inputs.download-parallel }}" > conan-settings/global.conf | ||
| conan config install conan-settings --type dir | ||
| rm -rf conan-settings | ||
|
|
||
| conan remote add -f conan-prerelease "${{ inputs.prerelease-remote-url }}" | ||
| conan remote list | ||
|
|
||
| - name: Install CI Conan lockfile | ||
| if: ${{ inputs.use-lockfile == 'true' }} | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| LOCK_DIR="${{ inputs.lockfile-dir }}" | ||
| [[ -f "${LOCK_DIR}/conan.lock" ]] || { | ||
| echo "Missing ${LOCK_DIR}/conan.lock (generate from pkg.lock)" | ||
| exit 1 | ||
| } | ||
| [[ -f "${LOCK_DIR}/conanfile.txt" ]] || { | ||
| echo "Missing ${LOCK_DIR}/conanfile.txt" | ||
| exit 1 | ||
| } | ||
| [[ -f "${LOCK_DIR}/pkg.lock" ]] || { | ||
| echo "Missing ${LOCK_DIR}/pkg.lock" | ||
| exit 1 | ||
| } | ||
|
|
||
| echo "Installing locked packages from ${LOCK_DIR}/conan.lock" | ||
| conan install "${LOCK_DIR}" \ | ||
| --lockfile="${LOCK_DIR}/conan.lock" \ | ||
| -r conan-prerelease | ||
|
|
||
| ABS_PKG_LOCK="$(cd "${LOCK_DIR}" && pwd)/pkg.lock" | ||
| echo "CI_PKG_LOCK=${ABS_PKG_LOCK}" >> "${GITHUB_ENV}" | ||
| echo "Using CI pkg.lock for slt: ${ABS_PKG_LOCK}" | ||
|
|
||
| - name: Install Matter app package | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| # Locked CI: do not re-float deps after conan --lockfile install. | ||
| # Nightly (use-lockfile=false): allow normal resolve. | ||
| CHECK_UPDATES="true" | ||
| if [[ "${{ inputs.use-lockfile }}" == "true" ]]; then | ||
| CHECK_UPDATES="false" | ||
| fi | ||
| slt install "matter_app/${{ inputs.package-version }}@silabs" -e conan \ | ||
| --check-updates="${CHECK_UPDATES}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,193 @@ | ||
| # CI package dependency locks | ||
|
|
||
| This directory holds **CI-only** dependency pins for Matter package validation | ||
| workflows (`.github/workflows/matter-packages-validation.yaml` and the package | ||
| builders it calls). | ||
|
|
||
| These files are **not** part of the `matter` / `matter_app` Conan packages. | ||
| Packaging pulls SLC/SDK trees and explicit recipe roots; it does not ship | ||
| `.github/`. | ||
|
|
||
| ## Why this exists | ||
|
|
||
| Package CI depends on many Silabs Conan packages (direct and transitive). | ||
| Floating ranges (for example in `slc/script/dependency_versions.yaml` / | ||
| generated `pkg.slt`) can pick up newly published prereleases and break PR CI | ||
| even when Matter code did not change. | ||
|
|
||
| This directory stores a **certified pin set**: | ||
|
|
||
| 1. **PR CI** installs that pin set so builds are reproducible. | ||
| 2. **Nightly CI** resolves freely with `slt install`, and on success refreshes | ||
| the pins so the next day’s PRs pick up a validated set. | ||
|
|
||
| ## Files | ||
|
|
||
| | File | Format | Role | | ||
| | --- | --- | --- | | ||
| | `pkg.lock` | SLT TOML-like lock | **Source of truth.** Exact pins from `slt install` (versions + Conan recipe revisions, plus archive tools). | | ||
| | `conan.lock` | Conan 2 JSON lock (`version: "0.5"`) | **Derived.** Used by `conan install --lockfile=...`. | | ||
| | `conanfile.txt` | Conan requires list | **Derived.** Companion graph for `conan install <dir> --lockfile=...`. | | ||
| | `README.md` | Markdown | This document. | | ||
|
|
||
| ### Important: two lock formats | ||
|
|
||
| | | `pkg.lock` | `conan.lock` | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I understand the role of these two files, but can you clarify the need for
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this provides the recipe for conan install
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is that not what
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No recipe → Conan doesn’t know what install you are doing. |
||
| | --- | --- | --- | | ||
| | Consumer | `slt install` | `conan install --lockfile` | | ||
| | Shape | `[dependency]` entries with `installer`, `type`, `ref` | JSON `requires` array of `name/version@user#revision` | | ||
| | Compatible with each other? | **No** — do not pass `pkg.lock` to `--lockfile` | | ||
|
|
||
| Always treat `pkg.lock` as the source and regenerate Conan files from it. | ||
|
|
||
| ## Source of truth and conversion | ||
|
|
||
| ```text | ||
| slt install --> pkg.lock --> pkg_lock_to_conan_lock.py --> conan.lock | ||
| \--> conanfile.txt | ||
| ``` | ||
|
|
||
| Converter script (repo root): | ||
|
|
||
| ```bash | ||
| python3 slc/script/pkg_lock_to_conan_lock.py \ | ||
| --input .github/pkg-manifest/pkg.lock \ | ||
| --output .github/pkg-manifest/conan.lock \ | ||
| --conanfile .github/pkg-manifest/conanfile.txt | ||
| ``` | ||
|
|
||
| What the converter does: | ||
|
|
||
| - Keeps only entries with `installer = "conan"`. | ||
| - Skips archive/tool packages (`commander`, `slc-cli`, `python`, `java21`, `zap`, …). | ||
| - By default **omits** `matter` and `matter_app` so CI can install PR-built | ||
| Matter packages on top of the locked third-party set. | ||
| - Prefers each entry’s `ref = "name/version@silabs#revision"` when present. | ||
| - Adds Silabs recipe `python_requires` (default | ||
| `silabs_package_assistant/1.8.0@silabs#...`) — required by Conan 2 lock | ||
| installs; not present in SLT `pkg.lock`. | ||
| - Writes a minimal `conanfile.txt` with exact `name/version@silabs` requires | ||
| (revision stripped in the requires list; revisions stay in `conan.lock`). | ||
|
|
||
| Optional flag: | ||
|
|
||
| ```bash | ||
| # Include matter/matter_app in the Conan lock (normally not needed in CI) | ||
| python3 slc/script/pkg_lock_to_conan_lock.py --include-matter ... | ||
| ``` | ||
|
|
||
| ## PR CI (locked / stable) | ||
|
|
||
| Triggered on pull requests into `main` / `release_*` with `use_lockfile=true` | ||
| (default). | ||
|
|
||
| Flow (via `.github/actions/install-pkg-manifest` + `packages/build_app.sh`): | ||
|
|
||
| 1. Restore the Actions cache that contains the PR’s exported `matter` / | ||
| `matter_app` packages. | ||
| 2. Configure the Conan remote (`conan-prerelease`). | ||
| 3. Install third-party deps: | ||
|
|
||
| ```bash | ||
| conan install .github/pkg-manifest \ | ||
| --lockfile=.github/pkg-manifest/conan.lock \ | ||
| -r conan-prerelease | ||
| ``` | ||
|
|
||
| 4. Install the PR Matter app package (no float after the lock install): | ||
|
|
||
| ```bash | ||
| slt install "matter_app/<PR_VERSION>@silabs" -e conan --check-updates=false | ||
| ``` | ||
|
|
||
| 5. Export `CI_PKG_LOCK` to the absolute path of `pkg.lock`. | ||
| 6. For each app build, `build_app.sh`: | ||
| - Copies `CI_PKG_LOCK` into the app directory as `pkg.lock`. | ||
| - Rewrites `matter` / `matter_app` entries to the PR package version | ||
| (drops recipe revision so local `export-pkg` packages resolve). | ||
| - Runs `slt install --check-updates=false` so SLT does not float. | ||
|
|
||
| Result: third-party packages stay at the certified pins; only Matter code under | ||
| test comes from the PR. | ||
|
|
||
| ## Nightly CI (resolve + refresh) | ||
|
|
||
| Lock refresh runs only on `release_2.10-1.6.1`: | ||
|
|
||
| - Nightly dispatch from `.github/workflows/trigger-matter-packages-validation.yaml` | ||
| with `use_lockfile=false`. | ||
| - A push to `release_2.10-1.6.1`. | ||
|
|
||
| Pushes to other branches (including test branches) install the existing locks | ||
| and do not open a refresh pull request. | ||
|
|
||
| Flow: | ||
|
|
||
| 1. **Build** without installing `.github/pkg-manifest` locks. | ||
| 2. App builds use normal `slt install` (`CHECK_UPDATES=true`) so newly published | ||
| packages matching `pkg.slt` ranges can be pulled. | ||
| 3. If the package app builds (and OTA check, when enabled) succeed, the | ||
| **Update CI pkg.lock and conan.lock** job: | ||
| - Runs unlocked `slt install` in `slc/apps/lock_app/wifi`. | ||
| - Copies the produced `pkg.lock` to `.github/pkg-manifest/pkg.lock`. | ||
| - Runs `pkg_lock_to_conan_lock.py` to refresh `conan.lock` and | ||
| `conanfile.txt`. | ||
| - Opens a pull request (via `silabs-matter-ci-bot`) with the three files | ||
| against the dispatched branch (for example `release_2.10-1.6.1`). | ||
| Reuses `automation/pkg-manifest-lock-<branch>` if a refresh PR is already open. | ||
|
|
||
| If nightly fails, the previous committed locks are left unchanged, so PRs keep | ||
| using the last green pin set. | ||
|
|
||
| ## Manual update (local) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there a use case for manual updates at all? I would assume that if we want to update package versions, we should do so by running full CI and allowing automation to update only if CI passes with new packages to ensure dependency bumps don't break anything.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is drafted for any corner case if needed. even if we do the changes manually, they can be only merged after running the CI. only the update part instead done by nightly can be done by maintaner. |
||
|
|
||
| Use this when you want to refresh pins without waiting for nightly: | ||
|
|
||
| ```bash | ||
| # From repo root, with slt/conan configured and remotes available | ||
| python3 slc/script/generate_pkg_slt.py -d slc # if pkg.slt missing | ||
| rm -f slc/apps/lock_app/wifi/pkg.lock | ||
| ( cd slc/apps/lock_app/wifi && slt install --check-updates=true ) | ||
|
|
||
| mkdir -p .github/pkg-manifest | ||
| cp -f slc/apps/lock_app/wifi/pkg.lock .github/pkg-manifest/pkg.lock | ||
|
|
||
| python3 slc/script/pkg_lock_to_conan_lock.py \ | ||
| --input .github/pkg-manifest/pkg.lock \ | ||
| --output .github/pkg-manifest/conan.lock \ | ||
| --conanfile .github/pkg-manifest/conanfile.txt | ||
|
|
||
| # Review diff, then commit all three files together | ||
| git add .github/pkg-manifest/pkg.lock \ | ||
| .github/pkg-manifest/conan.lock \ | ||
| .github/pkg-manifest/conanfile.txt | ||
| ``` | ||
|
|
||
| Always convert after editing or replacing `pkg.lock`. Never hand-edit | ||
| `conan.lock` alone unless you also update `pkg.lock`. | ||
|
|
||
| ## Secrets / remotes | ||
|
|
||
| Remote used by CI (same as `create-and-cache-matter-packages` / `install-tools`; no Artifactory login): | ||
|
|
||
| ```bash | ||
| conan remote add -f conan-prerelease https://conan-prerelease.silabs.net/ | ||
| ``` | ||
|
|
||
| ## Related code | ||
|
|
||
| | Path | Role | | ||
| | --- | --- | | ||
| | `slc/script/pkg_lock_to_conan_lock.py` | `pkg.lock` → `conan.lock` + `conanfile.txt` | | ||
| | `.github/actions/install-pkg-manifest/` | Restore Matter cache; locked Conan install; set `CI_PKG_LOCK` | | ||
| | `packages/build_app.sh` | Honors `CI_PKG_LOCK` / `--pkg-lock` and `CHECK_UPDATES` | | ||
| | `.github/workflows/matter-packages-validation.yaml` | PR vs nightly mode; nightly lock refresh job | | ||
| | `.github/workflows/trigger-matter-packages-validation.yaml` | Daily dispatch with `use_lockfile=false` | | ||
| | `slc/script/dependency_versions.yaml` | Floating ranges for local/dev `pkg.slt` generation (not the CI pin set) | | ||
|
|
||
| ## Seed note | ||
|
|
||
| The initial `pkg.lock` was taken from `slc/apps/lock_app/wifi/pkg.lock` (a | ||
| representative Wi‑Fi app resolve that also includes shared Thread/platform | ||
| packages). Nightly (or a manual refresh) should replace it with a newly | ||
| resolved lock after a green package validation run. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| { | ||
| "version": "0.5", | ||
| "requires": [ | ||
| "zigbee/9.2.0-alpha.46@silabs#8b6c495feeff4aae923ab949e23740788e9f8eb7", | ||
| "wiseconnect_common/4.2.0-alpha.66@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "wifi/4.2.0-alpha.146@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "usb/1.8.0@silabs#2d1c3440a76741001090a046509994a4", | ||
| "unity_test_framework/1.1.0@silabs#f418a9460d44f860469de8d9aba3d723", | ||
| "thread/3.2.0-alpha.45@silabs#8cce3fd4cfd5bd014a36f01905c4b0b9a111a5aa", | ||
| "siwx91x_nwp_interface/2.17.0-alpha.42@silabs#8255e4021015a2cba62472ae5f002ac13d5ed684", | ||
| "silabs_lwip/1.2.0-alpha.2@silabs#f5f97a2aff5c62c73de377fec9ba8122bf722831", | ||
| "segger/1.2.0@silabs#36f8201f19c01e288b5be67fda8f500d", | ||
| "security_tfm/1.2.0@silabs#0943f3f4ec758de891b2468ce589d7f3", | ||
| "security_sxsymcrypt/1.2.0@silabs#f242692bb498dc0e96c0544b5d61c41e", | ||
| "security_libcryptosoc/1.1.0@silabs#a41bcce963865004cdd183d9f003fd37", | ||
| "rail_library/3.2.0-alpha.120@silabs#1912fd314134738688a6dc37f9e0e8d0fb0330f2", | ||
| "printf/1.2.0@silabs#7b24522f4c76f3ef6865b3fe5cb974b0", | ||
| "platform_siwx91x/4.2.0-alpha.147@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "platform_core/1.2.0@silabs#2f3f6a843280008b0111d23b638daa76", | ||
| "platform/6.2.0-alpha.60@silabs#3994f55b7c6ffc12bff64bfde70fe671", | ||
| "paho_mqtt_embedded/1.0.0-alpha.7@silabs#f50ba31e06d951edb072ff8c0e3fa32e3ab5af4f", | ||
| "openthread_forks/3.2.0-alpha.45@silabs#8cce3fd4cfd5bd014a36f01905c4b0b9a111a5aa", | ||
| "ninja/1.12.1@silabs#9ce59e07a142d4da1c532b49742fe16d", | ||
| "network_stack_sdk/1.0.0-alpha.47@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "network_stack/1.0.0-alpha.127@silabs#3afebe8d74635276b58ad9a8c53d71a5b2b2156d", | ||
| "multiprotocol/1.2.0-alpha.42@silabs#647fb0ed26ba9c87ec2ed3da4bd9693f4ab4e91f", | ||
| "micriumos/6.2.0@silabs#236252f6e565c572a2e9a819050e153d", | ||
| "mbedtls/1.2.0@silabs#34f8e2586570896f0be441cc56a4a20d", | ||
| "llvm-arm-toolchain-for-embedded/21.1.1@silabs#877f3e0faf9658c618b4c2fe80bbb9fa", | ||
| "json_parser/1.1.0-alpha.90@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "iot_socket/1.0.0-alpha.84@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "gcc-arm-none-eabi/14.2.rel1@silabs#7a5e7220af325e610937338dde99651b", | ||
| "freertos/1.2.0@silabs#845766a5077fb17fdf2854ad1786079d", | ||
| "delta_dfu/1.1.0@silabs#1ebba4a316f35b36228125a8942d772c72943ec6", | ||
| "coremark/1.1.0@silabs#fad338900822dd7b4dd280d6713266f1", | ||
| "common_15_4/1.2.0-alpha.8@silabs#647fb0ed26ba9c87ec2ed3da4bd9693f4ab4e91f", | ||
| "cmsis/1.2.0@silabs#f8ebee69efcf85687bb21ce7a9d725ce", | ||
| "cmake/3.30.2@silabs#8798657aa78b2da38a3ae465f2d10090", | ||
| "bootloader/3.4.0@silabs#39519d377fcced2d472ecba7ff6869ca", | ||
| "boards/1.2.0@silabs#25a23460427c045026202be2911fb2a3", | ||
| "bluetooth_le_siwx91x/4.2.0-alpha.147@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "bluetooth_le_host/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7", | ||
| "bluetooth_le_controller/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7", | ||
| "bluetooth_common/1.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7", | ||
| "bluetooth_bgbuild/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7", | ||
| "bgapi_protocol/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7", | ||
| "base_sdk/2026.12.0@silabs#716052badc1ee855a7abbf99407f2d41", | ||
| "azure_freertos_middleware/1.1.0-alpha.71@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84", | ||
| "aws_sdk/3.1.0-alpha.92@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84" | ||
| ], | ||
| "build_requires": [ | ||
| "ninja/1.12.1@silabs#9ce59e07a142d4da1c532b49742fe16d", | ||
| "llvm-arm-toolchain-for-embedded/21.1.1@silabs#877f3e0faf9658c618b4c2fe80bbb9fa", | ||
| "gcc-arm-none-eabi/14.2.rel1@silabs#7a5e7220af325e610937338dde99651b", | ||
| "cmake/3.30.2@silabs#8798657aa78b2da38a3ae465f2d10090", | ||
| "bluetooth_bgbuild/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7" | ||
| ], | ||
| "python_requires": [ | ||
| "silabs_package_assistant/1.8.0@silabs#1f4a525f99fdcc0d289cfd2fd532b6aefa7006c7" | ||
| ], | ||
| "config_requires": [] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for my understanding why do we
conan installall other packages, andslt installonlymatter_apppackage? can we perhaps consolidate onslt install <package> -e conanfor everything for consistency?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure let me give it a try