Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions .github/actions/install-pkg-manifest/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
name: Install package manifest
description: >
Restore Matter packages. In locked mode, install CI Conan lock derived from
SLT pkg.lock, then install local matter_app. Also exports CI_PKG_LOCK for
build_app.sh (slt --check-updates=false).

inputs:
package-version:
description: Version of matter_app to install from the restored Matter cache
required: true
conan-cache-key:
description: Cache key created by the Matter package producer
required: true
use-lockfile:
description: If true, conan install --lockfile + export CI_PKG_LOCK
required: false
default: "true"
fail-on-cache-miss:
description: Fail when the Matter package cache does not exist
required: false
default: "true"
download-parallel:
description: Maximum parallel Conan downloads after restore
required: false
default: "1"
lockfile-dir:
description: Directory with pkg.lock, conan.lock, conanfile.txt
required: false
default: .github/pkg-manifest
prerelease-remote-url:
description: Conan prerelease remote URL
required: false
default: https://conan-prerelease.silabs.net/

outputs:
cache-hit:
description: Whether the Matter package cache was restored
value: ${{ steps.restore.outputs.cache-hit }}

runs:
using: composite
steps:
- name: Prepare Conan home
shell: bash
run: |
set -euo pipefail
: "${CONAN_HOME:?CONAN_HOME must be set (run cache-tools first)}"
rm -rf "${CONAN_HOME}"
mkdir -p "${CONAN_HOME}"

- name: Restore Matter package cache
id: restore
uses: actions/cache/restore@v6
with:
path: ${{ env.CONAN_HOME }}
key: ${{ inputs.conan-cache-key }}
fail-on-cache-miss: ${{ inputs.fail-on-cache-miss }}

- name: Configure Conan
shell: bash
run: |
set -euo pipefail
mkdir -p conan-settings
printf 'core.download:parallel=%s\n' "${{ inputs.download-parallel }}" > conan-settings/global.conf
conan config install conan-settings --type dir
rm -rf conan-settings

conan remote add -f conan-prerelease "${{ inputs.prerelease-remote-url }}"
conan remote list

- name: Install CI Conan lockfile
if: ${{ inputs.use-lockfile == 'true' }}
shell: bash
run: |
set -euo pipefail
LOCK_DIR="${{ inputs.lockfile-dir }}"
[[ -f "${LOCK_DIR}/conan.lock" ]] || {
echo "Missing ${LOCK_DIR}/conan.lock (generate from pkg.lock)"
exit 1
}
[[ -f "${LOCK_DIR}/conanfile.txt" ]] || {
echo "Missing ${LOCK_DIR}/conanfile.txt"
exit 1
}
[[ -f "${LOCK_DIR}/pkg.lock" ]] || {
echo "Missing ${LOCK_DIR}/pkg.lock"
exit 1
}

echo "Installing locked packages from ${LOCK_DIR}/conan.lock"
conan install "${LOCK_DIR}" \
--lockfile="${LOCK_DIR}/conan.lock" \
-r conan-prerelease

ABS_PKG_LOCK="$(cd "${LOCK_DIR}" && pwd)/pkg.lock"
echo "CI_PKG_LOCK=${ABS_PKG_LOCK}" >> "${GITHUB_ENV}"
echo "Using CI pkg.lock for slt: ${ABS_PKG_LOCK}"

- name: Install Matter app package
shell: bash
run: |
set -euo pipefail
# Locked CI: do not re-float deps after conan --lockfile install.
# Nightly (use-lockfile=false): allow normal resolve.
CHECK_UPDATES="true"
if [[ "${{ inputs.use-lockfile }}" == "true" ]]; then
CHECK_UPDATES="false"
fi
slt install "matter_app/${{ inputs.package-version }}@silabs" -e conan \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for my understanding why do we conan install all other packages, and slt install only matter_app package? can we perhaps consolidate on slt install <package> -e conan for everything for consistency?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure let me give it a try

--check-updates="${CHECK_UPDATES}"
193 changes: 193 additions & 0 deletions .github/pkg-manifest/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# CI package dependency locks

This directory holds **CI-only** dependency pins for Matter package validation
workflows (`.github/workflows/matter-packages-validation.yaml` and the package
builders it calls).

These files are **not** part of the `matter` / `matter_app` Conan packages.
Packaging pulls SLC/SDK trees and explicit recipe roots; it does not ship
`.github/`.

## Why this exists

Package CI depends on many Silabs Conan packages (direct and transitive).
Floating ranges (for example in `slc/script/dependency_versions.yaml` /
generated `pkg.slt`) can pick up newly published prereleases and break PR CI
even when Matter code did not change.

This directory stores a **certified pin set**:

1. **PR CI** installs that pin set so builds are reproducible.
2. **Nightly CI** resolves freely with `slt install`, and on success refreshes
the pins so the next day’s PRs pick up a validated set.

## Files

| File | Format | Role |
| --- | --- | --- |
| `pkg.lock` | SLT TOML-like lock | **Source of truth.** Exact pins from `slt install` (versions + Conan recipe revisions, plus archive tools). |
| `conan.lock` | Conan 2 JSON lock (`version: "0.5"`) | **Derived.** Used by `conan install --lockfile=...`. |
| `conanfile.txt` | Conan requires list | **Derived.** Companion graph for `conan install <dir> --lockfile=...`. |
| `README.md` | Markdown | This document. |

### Important: two lock formats

| | `pkg.lock` | `conan.lock` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the role of these two files, but can you clarify the need for conanfile.txt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this provides the recipe for conan install

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that not what conan.lock does?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No recipe → Conan doesn’t know what install you are doing.
Recipe + no lock → Conan resolves versions (can float / change over time).
Recipe + lock → Conan installs that same graph, pinned exactly

| --- | --- | --- |
| Consumer | `slt install` | `conan install --lockfile` |
| Shape | `[dependency]` entries with `installer`, `type`, `ref` | JSON `requires` array of `name/version@user#revision` |
| Compatible with each other? | **No** — do not pass `pkg.lock` to `--lockfile` |

Always treat `pkg.lock` as the source and regenerate Conan files from it.

## Source of truth and conversion

```text
slt install --> pkg.lock --> pkg_lock_to_conan_lock.py --> conan.lock
\--> conanfile.txt
```

Converter script (repo root):

```bash
python3 slc/script/pkg_lock_to_conan_lock.py \
--input .github/pkg-manifest/pkg.lock \
--output .github/pkg-manifest/conan.lock \
--conanfile .github/pkg-manifest/conanfile.txt
```

What the converter does:

- Keeps only entries with `installer = "conan"`.
- Skips archive/tool packages (`commander`, `slc-cli`, `python`, `java21`, `zap`, …).
- By default **omits** `matter` and `matter_app` so CI can install PR-built
Matter packages on top of the locked third-party set.
- Prefers each entry’s `ref = "name/version@silabs#revision"` when present.
- Adds Silabs recipe `python_requires` (default
`silabs_package_assistant/1.8.0@silabs#...`) — required by Conan 2 lock
installs; not present in SLT `pkg.lock`.
- Writes a minimal `conanfile.txt` with exact `name/version@silabs` requires
(revision stripped in the requires list; revisions stay in `conan.lock`).

Optional flag:

```bash
# Include matter/matter_app in the Conan lock (normally not needed in CI)
python3 slc/script/pkg_lock_to_conan_lock.py --include-matter ...
```

## PR CI (locked / stable)

Triggered on pull requests into `main` / `release_*` with `use_lockfile=true`
(default).

Flow (via `.github/actions/install-pkg-manifest` + `packages/build_app.sh`):

1. Restore the Actions cache that contains the PR’s exported `matter` /
`matter_app` packages.
2. Configure the Conan remote (`conan-prerelease`).
3. Install third-party deps:

```bash
conan install .github/pkg-manifest \
--lockfile=.github/pkg-manifest/conan.lock \
-r conan-prerelease
```

4. Install the PR Matter app package (no float after the lock install):

```bash
slt install "matter_app/<PR_VERSION>@silabs" -e conan --check-updates=false
```

5. Export `CI_PKG_LOCK` to the absolute path of `pkg.lock`.
6. For each app build, `build_app.sh`:
- Copies `CI_PKG_LOCK` into the app directory as `pkg.lock`.
- Rewrites `matter` / `matter_app` entries to the PR package version
(drops recipe revision so local `export-pkg` packages resolve).
- Runs `slt install --check-updates=false` so SLT does not float.

Result: third-party packages stay at the certified pins; only Matter code under
test comes from the PR.

## Nightly CI (resolve + refresh)

Lock refresh runs only on `release_2.10-1.6.1`:

- Nightly dispatch from `.github/workflows/trigger-matter-packages-validation.yaml`
with `use_lockfile=false`.
- A push to `release_2.10-1.6.1`.

Pushes to other branches (including test branches) install the existing locks
and do not open a refresh pull request.

Flow:

1. **Build** without installing `.github/pkg-manifest` locks.
2. App builds use normal `slt install` (`CHECK_UPDATES=true`) so newly published
packages matching `pkg.slt` ranges can be pulled.
3. If the package app builds (and OTA check, when enabled) succeed, the
**Update CI pkg.lock and conan.lock** job:
- Runs unlocked `slt install` in `slc/apps/lock_app/wifi`.
- Copies the produced `pkg.lock` to `.github/pkg-manifest/pkg.lock`.
- Runs `pkg_lock_to_conan_lock.py` to refresh `conan.lock` and
`conanfile.txt`.
- Opens a pull request (via `silabs-matter-ci-bot`) with the three files
against the dispatched branch (for example `release_2.10-1.6.1`).
Reuses `automation/pkg-manifest-lock-<branch>` if a refresh PR is already open.

If nightly fails, the previous committed locks are left unchanged, so PRs keep
using the last green pin set.

## Manual update (local)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a use case for manual updates at all? I would assume that if we want to update package versions, we should do so by running full CI and allowing automation to update only if CI passes with new packages to ensure dependency bumps don't break anything.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is drafted for any corner case if needed. even if we do the changes manually, they can be only merged after running the CI. only the update part instead done by nightly can be done by maintaner.
like example Nightly is down due to some infra issue and we know the particular build is stable etc. just for a rare corner case


Use this when you want to refresh pins without waiting for nightly:

```bash
# From repo root, with slt/conan configured and remotes available
python3 slc/script/generate_pkg_slt.py -d slc # if pkg.slt missing
rm -f slc/apps/lock_app/wifi/pkg.lock
( cd slc/apps/lock_app/wifi && slt install --check-updates=true )

mkdir -p .github/pkg-manifest
cp -f slc/apps/lock_app/wifi/pkg.lock .github/pkg-manifest/pkg.lock

python3 slc/script/pkg_lock_to_conan_lock.py \
--input .github/pkg-manifest/pkg.lock \
--output .github/pkg-manifest/conan.lock \
--conanfile .github/pkg-manifest/conanfile.txt

# Review diff, then commit all three files together
git add .github/pkg-manifest/pkg.lock \
.github/pkg-manifest/conan.lock \
.github/pkg-manifest/conanfile.txt
```

Always convert after editing or replacing `pkg.lock`. Never hand-edit
`conan.lock` alone unless you also update `pkg.lock`.

## Secrets / remotes

Remote used by CI (same as `create-and-cache-matter-packages` / `install-tools`; no Artifactory login):

```bash
conan remote add -f conan-prerelease https://conan-prerelease.silabs.net/
```

## Related code

| Path | Role |
| --- | --- |
| `slc/script/pkg_lock_to_conan_lock.py` | `pkg.lock` → `conan.lock` + `conanfile.txt` |
| `.github/actions/install-pkg-manifest/` | Restore Matter cache; locked Conan install; set `CI_PKG_LOCK` |
| `packages/build_app.sh` | Honors `CI_PKG_LOCK` / `--pkg-lock` and `CHECK_UPDATES` |
| `.github/workflows/matter-packages-validation.yaml` | PR vs nightly mode; nightly lock refresh job |
| `.github/workflows/trigger-matter-packages-validation.yaml` | Daily dispatch with `use_lockfile=false` |
| `slc/script/dependency_versions.yaml` | Floating ranges for local/dev `pkg.slt` generation (not the CI pin set) |

## Seed note

The initial `pkg.lock` was taken from `slc/apps/lock_app/wifi/pkg.lock` (a
representative Wi‑Fi app resolve that also includes shared Thread/platform
packages). Nightly (or a manual refresh) should replace it with a newly
resolved lock after a green package validation run.
62 changes: 62 additions & 0 deletions .github/pkg-manifest/conan.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"version": "0.5",
"requires": [
"zigbee/9.2.0-alpha.46@silabs#8b6c495feeff4aae923ab949e23740788e9f8eb7",
"wiseconnect_common/4.2.0-alpha.66@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"wifi/4.2.0-alpha.146@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"usb/1.8.0@silabs#2d1c3440a76741001090a046509994a4",
"unity_test_framework/1.1.0@silabs#f418a9460d44f860469de8d9aba3d723",
"thread/3.2.0-alpha.45@silabs#8cce3fd4cfd5bd014a36f01905c4b0b9a111a5aa",
"siwx91x_nwp_interface/2.17.0-alpha.42@silabs#8255e4021015a2cba62472ae5f002ac13d5ed684",
"silabs_lwip/1.2.0-alpha.2@silabs#f5f97a2aff5c62c73de377fec9ba8122bf722831",
"segger/1.2.0@silabs#36f8201f19c01e288b5be67fda8f500d",
"security_tfm/1.2.0@silabs#0943f3f4ec758de891b2468ce589d7f3",
"security_sxsymcrypt/1.2.0@silabs#f242692bb498dc0e96c0544b5d61c41e",
"security_libcryptosoc/1.1.0@silabs#a41bcce963865004cdd183d9f003fd37",
"rail_library/3.2.0-alpha.120@silabs#1912fd314134738688a6dc37f9e0e8d0fb0330f2",
"printf/1.2.0@silabs#7b24522f4c76f3ef6865b3fe5cb974b0",
"platform_siwx91x/4.2.0-alpha.147@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"platform_core/1.2.0@silabs#2f3f6a843280008b0111d23b638daa76",
"platform/6.2.0-alpha.60@silabs#3994f55b7c6ffc12bff64bfde70fe671",
"paho_mqtt_embedded/1.0.0-alpha.7@silabs#f50ba31e06d951edb072ff8c0e3fa32e3ab5af4f",
"openthread_forks/3.2.0-alpha.45@silabs#8cce3fd4cfd5bd014a36f01905c4b0b9a111a5aa",
"ninja/1.12.1@silabs#9ce59e07a142d4da1c532b49742fe16d",
"network_stack_sdk/1.0.0-alpha.47@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"network_stack/1.0.0-alpha.127@silabs#3afebe8d74635276b58ad9a8c53d71a5b2b2156d",
"multiprotocol/1.2.0-alpha.42@silabs#647fb0ed26ba9c87ec2ed3da4bd9693f4ab4e91f",
"micriumos/6.2.0@silabs#236252f6e565c572a2e9a819050e153d",
"mbedtls/1.2.0@silabs#34f8e2586570896f0be441cc56a4a20d",
"llvm-arm-toolchain-for-embedded/21.1.1@silabs#877f3e0faf9658c618b4c2fe80bbb9fa",
"json_parser/1.1.0-alpha.90@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"iot_socket/1.0.0-alpha.84@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"gcc-arm-none-eabi/14.2.rel1@silabs#7a5e7220af325e610937338dde99651b",
"freertos/1.2.0@silabs#845766a5077fb17fdf2854ad1786079d",
"delta_dfu/1.1.0@silabs#1ebba4a316f35b36228125a8942d772c72943ec6",
"coremark/1.1.0@silabs#fad338900822dd7b4dd280d6713266f1",
"common_15_4/1.2.0-alpha.8@silabs#647fb0ed26ba9c87ec2ed3da4bd9693f4ab4e91f",
"cmsis/1.2.0@silabs#f8ebee69efcf85687bb21ce7a9d725ce",
"cmake/3.30.2@silabs#8798657aa78b2da38a3ae465f2d10090",
"bootloader/3.4.0@silabs#39519d377fcced2d472ecba7ff6869ca",
"boards/1.2.0@silabs#25a23460427c045026202be2911fb2a3",
"bluetooth_le_siwx91x/4.2.0-alpha.147@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"bluetooth_le_host/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7",
"bluetooth_le_controller/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7",
"bluetooth_common/1.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7",
"bluetooth_bgbuild/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7",
"bgapi_protocol/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7",
"base_sdk/2026.12.0@silabs#716052badc1ee855a7abbf99407f2d41",
"azure_freertos_middleware/1.1.0-alpha.71@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84",
"aws_sdk/3.1.0-alpha.92@silabs#52bec072c7581aa2e60fe3012d6e554b2b1cff84"
],
"build_requires": [
"ninja/1.12.1@silabs#9ce59e07a142d4da1c532b49742fe16d",
"llvm-arm-toolchain-for-embedded/21.1.1@silabs#877f3e0faf9658c618b4c2fe80bbb9fa",
"gcc-arm-none-eabi/14.2.rel1@silabs#7a5e7220af325e610937338dde99651b",
"cmake/3.30.2@silabs#8798657aa78b2da38a3ae465f2d10090",
"bluetooth_bgbuild/11.2.0-alpha.69@silabs#91f8b6a6799e6a7e9cecc95fc836f3b6ea30c8e7"
],
"python_requires": [
"silabs_package_assistant/1.8.0@silabs#1f4a525f99fdcc0d289cfd2fd532b6aefa7006c7"
],
"config_requires": []
}
Loading
Loading