Skip to content

Commit 834fdc7

Browse files
Production-ready monorepo: all ROS 2 distributions, robostack-bot, CI (#5)
Ready for RoboStack: production defaults, package requests from anyone, import tool
2 parents bf2b9d1 + ba84c4b commit 834fdc7

50 files changed

Lines changed: 1907 additions & 387 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
name: robostack-debug-build
3+
description: Debug and fix a RoboStack package that fails to build (compile, link, configure, patch or test failures) for one ROS distribution. Use when a build-one / CI build of a ros2-<pkg> recipe fails.
4+
---
5+
6+
# Debug a failed package build
7+
8+
Work from the repository root; `<d>` is the distribution, `<pkg>` the conda name
9+
(`ros2-<ros-name-with-dashes>`).
10+
11+
## 1. Reproduce with one package
12+
13+
```bash
14+
pixi run rs <d> generate-recipes # if distros/<d>/work/recipes is missing or stale
15+
pixi run rs <d> build-one ros2-<pkg>
16+
```
17+
18+
`build-one` copies the package's patches into the recipe and builds without
19+
`--skip-existing`. For a CI failure, first read the failing job's log
20+
(`gh run view <run> --job <job> --log-failed`) and search for `× error`,
21+
`CMake Error`, `error:` and `undefined reference`.
22+
23+
## 2. Find the work directory
24+
25+
```bash
26+
ls -td distros/<d>/work/output/bld/rattler-build_ros2-<pkg>_*/work | head -1
27+
```
28+
29+
In it:
30+
- `conda_build.log`: the full log. Look for compile, link and configure errors,
31+
patch failures, missing files.
32+
- `build_env.sh`: `PREFIX` (host env), `BUILD_PREFIX`, `SRC_DIR`, `RECIPE_DIR`.
33+
- `.source_info.json` (`jq .`): the fetched source revision.
34+
35+
## 3. Investigate by failure class
36+
37+
| symptom | look at |
38+
|---|---|
39+
| missing header | `requirements.host` of the recipe; is it under `$PREFIX/include`? |
40+
| `Could not find a package configuration file provided by "X"` | is X in host? Does a conda-forge package provide that CMake config? A mapping in `shared/robostack.yaml` may be wrong, or vinca may have built a ROS package under the conda-forge name (see below) |
41+
| undefined symbols | host deps, `$PREFIX/lib`, link flags |
42+
| configure failure | flags in `conda_build.sh`; rerun with verbosity |
43+
| patch does not apply | refresh it, see the `robostack-patch` skill |
44+
| hard-coded prefixes / rpaths | relocatability; inspect with `otool -L` (macOS) / `patchelf --print-rpath` |
45+
46+
Reproduce interactively:
47+
48+
```bash
49+
cd <work-directory>
50+
source build_env.sh
51+
bash -x conda_build.sh 2>&1 | less
52+
```
53+
54+
## 4. Common fixes
55+
56+
- **Boost 1.88+**: needs C++14 or newer; replace `-std=c++11` / `CMAKE_CXX_STANDARD 11`.
57+
- **Python modules on macOS**: don't link `Python::Python`:
58+
```cmake
59+
if(APPLE)
60+
set_target_properties(${_name} PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
61+
else()
62+
target_link_libraries(${_name} ${PYTHON_LIBRARIES})
63+
endif()
64+
```
65+
- **Windows `min`/`max` macro clashes**: `target_compile_definitions(<target> PRIVATE NOMINMAX WIN32_LEAN_AND_MEAN)` under `if(WIN32)`.
66+
- **gtest / test failures**: add the missing dependency in `dependencies.yaml`, or
67+
disable the tests when safe; no custom shims.
68+
- **Qt plugins (rviz, rtabmap)**: make sure CMake finds the intended Qt major version.
69+
- **Missing dependency**: `shared/dependencies.yaml` (all distributions) or
70+
`distros/<d>/patch/dependencies.yaml` (one), keyed by ROS package name:
71+
```yaml
72+
<ros_pkg>:
73+
add_host: [<conda-pkg>]
74+
add_run: [<conda-pkg>]
75+
```
76+
- **A conda-forge package is shadowed**: when a rosdep key that is also a ROS package
77+
maps to a conda-forge package (e.g. `tl_expected` -> `cpp-expected`), vinca may build
78+
the ROS package under the conda-forge name. Add it to `packages_skip_by_deps`
79+
(`shared/vinca.yaml`), and evict the stale build from the PR cache
80+
(`evict_cache` in `distros/<d>/ci.yaml`, plain names work).
81+
- **CMake args for one package**: `additional_cmake_args` in `pkg_additional_info.yaml`.
82+
83+
## 5. Make it stick
84+
85+
1. Turn source edits into a patch (`robostack-patch` skill).
86+
2. Rebuild: `pixi run rs <d> build-one ros2-<pkg>`.
87+
3. If the fixed package was already built in the PR, add it to `evict_cache` in
88+
`distros/<d>/ci.yaml`. A package that is already published needs a new build
89+
number: `<ros_pkg>: {build_number: <global + 1>}` in `distros/<d>/pkg_additional_info.yaml`.
90+
4. Check whether the same fix applies to other distributions; port it only where the
91+
source is compatible.
92+
93+
Triage order with many failures: fix the easy ones first, move on once something gets
94+
complex, and come back after the queue is shorter.
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
name: robostack-package-selection
3+
description: Add, exclude or restrict ROS packages in RoboStack vinca.yaml files (shared/ and distros/<d>/), including platform-only packages, dependency skips and the build gap report. Use when a package should be added, removed, made Linux-only, or when recipes/artifacts don't line up.
4+
---
5+
6+
# Package selection (vinca.yaml)
7+
8+
`distros/<d>/work/vinca.yaml` is assembled from `shared/vinca.yaml` plus
9+
`distros/<d>/vinca.yaml`. The three package lists are concatenated; for other keys
10+
the distribution wins. A package selected for every distribution goes into
11+
`shared/vinca.yaml`.
12+
13+
## Add packages
14+
15+
```bash
16+
pixi run rs add-package foxglove_bridge rqt_gauges # every distribution that has them
17+
pixi run rs add-package foxglove_bridge humble jazzy --preview
18+
```
19+
20+
This is the bot's `add-package`. It inserts in sorted position, chooses `shared/` when
21+
every distribution gets the same packages, and lists what will be built on linux-64.
22+
Names are ROS package names; dashes and underscores both work.
23+
24+
## The three ways to exclude a package
25+
26+
vinca's revision is pinned in `pixi.toml`. If that pin moves, re-check
27+
`vinca/main.py` and `vinca/resolve.py`.
28+
29+
1. **`packages_select_by_deps` under a platform condition**: the primary way to keep a
30+
package's own recipe off a platform. Every listed name is selected
31+
unconditionally, so not listing it for a platform is what excludes it:
32+
```yaml
33+
packages_select_by_deps:
34+
- if: linux
35+
then:
36+
- some_linux_only_pkg
37+
```
38+
2. **`packages_skip_by_deps`**: only stops transitive pull-in (`ignore_pkgs` for
39+
`distro.get_depends()`). It doesn't stop a package listed directly in (1). It's
40+
also the tool for ROS packages that map to conda-forge in `shared/robostack.yaml`
41+
(`tl_expected`, `sophus`), so that vinca doesn't build them under the conda-forge
42+
name.
43+
3. **`packages_remove_from_deps`**: removes the package's own recipe *and* strips it
44+
from every other package's host/run dependencies, together and inseparably. Use
45+
it only if nothing selected legitimately needs it; otherwise use (1) + (2).
46+
47+
Keep (2) and (3) coherent with the platform conditions in (1). A truly platform-only
48+
package goes behind a condition, not into a comment.
49+
50+
## Recipes vs. artifacts
51+
52+
```bash
53+
pixi run rs <d> generate-recipes --platform <platform>
54+
pixi run rs <d> gap-report
55+
```
56+
57+
- **Built artifacts without a recipe directory**: packages in
58+
`distros/<d>/work/output/<platform>` that aren't selected any more. Add their seeds,
59+
or accept that they are gone.
60+
- **Recipe directories without an artifact**: generated recipes still to build on this
61+
platform.
62+
63+
Regenerate recipes after editing `vinca.yaml` before expecting the report to change.
64+
65+
## rosdep mappings
66+
67+
`shared/robostack.yaml` maps rosdep keys to conda packages (per platform);
68+
`shared/packages-ignore.yaml` lists keys to drop. Changes there affect every
69+
distribution: check them with `generate-recipes` for at least two distributions.
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
name: robostack-patch
3+
description: Create, refresh, place, port and validate source patches for RoboStack ROS packages (distros/<d>/patch/). Use after editing sources in a rattler-build work directory, when a patch no longer applies, or when porting a fix to another distribution.
4+
---
5+
6+
# Patches
7+
8+
## Where they live
9+
10+
- `distros/<d>/patch/ros-<d>-<pkg>.patch` (all platforms), or with a platform suffix
11+
`.osx.patch`, `.linux.patch`, `.win.patch`, `.unix.patch`. `<pkg>` uses dashes.
12+
- One patch per package and platform. Merge new hunks into the existing file instead
13+
of adding variants.
14+
- vinca picks them up by name when generating recipes; nothing else needs wiring.
15+
`pixi run rs check` rejects names of other distributions.
16+
- `distros/<d>/patch/dependencies.yaml` is for dependency fixes, not source patches
17+
(`shared/dependencies.yaml` for all distributions).
18+
19+
## Create a patch from work-directory edits
20+
21+
```bash
22+
WORK=$(ls -td distros/<d>/work/output/bld/rattler-build_ros2-<pkg>_*/work | head -1)
23+
cd "$WORK"
24+
# edit the sources, rebuild with `bash conda_build.sh` (after `source build_env.sh`)
25+
rattler-build create-patch --directory . --name ros-<d>-<pkg> \
26+
--exclude "*.o,*.so,*.dylib,*.a,*.pyc,__pycache__,build/" --dry-run
27+
rattler-build create-patch --directory . --name ros-<d>-<pkg> \
28+
--exclude "*.o,*.so,*.dylib,*.a,*.pyc,__pycache__,build/"
29+
```
30+
31+
Move the result to `distros/<d>/patch/ros-<d>-<pkg>.patch`, merging it with an
32+
existing patch for that package if there is one. Then:
33+
34+
```bash
35+
pixi run rs <d> build-one ros2-<pkg>
36+
```
37+
38+
Without a work directory, a `git diff` against the source checkout from
39+
`.source_info.json` works too (paths relative to the source root, `a/` and `b/` prefixes).
40+
41+
## Check that patches apply
42+
43+
```bash
44+
pixi run rs <d> check-patches # every recipe with patches
45+
pixi run rs <d> check-patches --recipe ros2-<pkg> # one (repeat --recipe for more)
46+
pixi run rs <d> check-patches --dry --recipe ros2-<pkg> # prepare only
47+
pixi run rs <d> check-orphaned-patches # platform patches without a recipe
48+
```
49+
50+
## Refresh a patch that no longer applies
51+
52+
A new upstream version moved the code: build the package without the patch (rename it
53+
temporarily), re-apply the change by hand in the work directory, regenerate with
54+
`create-patch`, and check that the old intent is still needed. Upstream may have fixed
55+
it, in which case delete the patch.
56+
57+
## Port a fix to another distribution
58+
59+
- Port only to an existing package with a compatible source. Compare the versions in
60+
`distros/<d>/rosdistro_snapshot.yaml` and check with `check-patches`. Never copy a
61+
patch only because a file of that name exists elsewhere.
62+
- Rename to the target distribution (`ros-<other>-<pkg>.patch`).
63+
- After the PR's first build, add the package to `evict_cache` in the target's
64+
`ci.yaml` if CI already cached it.
65+
- Fixes that belong upstream: open a PR on the package's repository and note it in the
66+
patch header.
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
name: robostack-versions
3+
description: Update ROS package versions (rosdistro snapshot) and conda-forge pins for RoboStack distributions, bump build numbers and the distro mutex, plan full rebuilds, and find or rebuild packages built against outdated pins. Use for version updates, pinning/migration changes, pin conflicts, or full rebuilds.
4+
---
5+
6+
# Versions, pins and rebuilds
7+
8+
robostack-bot does the routine part: `update-rosdistro-snapshot` (daily, when a new
9+
rosdistro sync is tagged) and
10+
`update-conda-forge-pinning` (weekly, or on request) open PRs that already contain the
11+
bumps below. Do it by hand when the bot fails or something extra is needed.
12+
13+
## What is pinned where
14+
15+
- **ROS package versions**: `distros/<d>/rosdistro_snapshot.yaml`, plus
16+
`rosdistro_additional_recipes.yaml` for packages outside rosdistro.
17+
- **conda-forge pins**:
18+
- `shared/pinning/conda_forge.yaml` holds the pinning version and migrations for
19+
every distribution that doesn't set its own.
20+
- `shared/pinning/overrides.yaml` holds local pins.
21+
- `distros/<d>/distro.yaml` can set `conda_forge_pinning_version` together with
22+
`conda_forge_migrations` (its own pinning, e.g. until its next full rebuild), and
23+
`pinning_overrides` (replace shared pins by name).
24+
- `pixi run rs <d> render-pinning` writes the committed
25+
`distros/<d>/conda_build_config.yaml`.
26+
- **The mutex** (`mutex_package` in `distros/<d>/vinca.yaml`): `ros2-distro-mutex`
27+
with `upper_bound: x.x`. Packages built with mutex 0.11 can't be installed with
28+
0.10 builds. Its `run_constraints` pin the ABI-relevant libraries for users'
29+
environments and must agree with the rendered pins.
30+
31+
## Update the snapshot
32+
33+
```bash
34+
pixi run rs rosdistro-syncs # which distributions have a new sync (what the bot checks daily)
35+
pixi run rs <d> update-snapshot # = bot command: latest sync, version bumps, rebuild bump
36+
# or only the file, for a given sync tag (any ros/rosdistro ref) or rosdistro master:
37+
pixi run rs <d> create-snapshot --rosdistro-ref <d>/2026-10-05
38+
pixi run rs <d> create-snapshot
39+
```
40+
41+
- The ROS release team tags every sync from ros-testing to the main repositories in
42+
ros/rosdistro (`<distro>/<YYYY-MM-DD>`, announced on Discourse as "New Packages for
43+
..."). Snapshotting the tag ships what users get from the official repositories;
44+
rosdistro master also contains releases that are still only in ros-testing.
45+
- `distros/<d>/distro.yaml` records the sync as `rosdistro_sync`. `rosdistro_sync:
46+
manual` (rolling, which isn't tagged regularly) uses rosdistro master instead and
47+
isn't checked daily.
48+
- `update-snapshot` also performs the rebuild bump (next section).
49+
50+
## Update the conda-forge pinning
51+
52+
```bash
53+
pixi run rs update-pinning # = bot command, for all distributions that follow shared/
54+
pixi run rs <d> render-pinning # after editing shared/pinning or distro.yaml
55+
pixi run rs <d> check-deps # solve all non-ROS deps + mutex constraints, nothing built
56+
```
57+
58+
`check-deps` reports:
59+
- `PIN MISMATCH`: a mutex `run_constraints` entry contradicts the rendered pin. Fix
60+
`vinca.yaml` or the pinning; never `conda_build_config.yaml`.
61+
- Per conflicting package: the recipes that need it, the clashing pin, the solver
62+
explanation and the conda-forge migration status. "no migration" means the feedstock
63+
only needs a rebuild upstream.
64+
65+
Exit code 1 means something conflicts. Status of migrations:
66+
https://conda-forge.org/status/.
67+
68+
## Rebuild bump (full rebuild)
69+
70+
When the snapshot or the pins change, everything is rebuilt:
71+
72+
1. `build_number` in `distros/<d>/vinca.yaml` = the highest build number of the
73+
distribution's released packages + 1, so every package gets a new build.
74+
2. Mutex minor version + 1, e.g. `0.10.0` -> `0.11.0`. Nothing else hard-codes the
75+
mutex version.
76+
3. Remove all per-package `build_number` entries in `distros/<d>/pkg_additional_info.yaml`.
77+
4. Update mutex `run_constraints` to the new pins, then run `check-deps`.
78+
5. `pixi run rs check`.
79+
80+
The bot does 1–4. It only moves constraints of the form `<pkg> <version>.*`; ranges
81+
and other constraints are left for you to check.
82+
83+
In the PR, CI uses its cache unless told otherwise. A full rebuild PR sets
84+
`full_rebuild: true` in `distros/<d>/ci.yaml` (reset after merging).
85+
86+
## Rebuild a few packages
87+
88+
- Published already: `<ros_pkg>: {build_number: <global + 1>}` in
89+
`distros/<d>/pkg_additional_info.yaml`. Shared settings stay in
90+
`shared/pkg_additional_info.yaml`; the distribution's keys win.
91+
- Only in the current PR: `evict_cache` in `distros/<d>/ci.yaml`.
92+
93+
## Packages built against outdated pins
94+
95+
```bash
96+
pixi run rs <d> find-stale # = bot find-stale-packages
97+
pixi run rs <d> check-deps --stale # local artifacts
98+
pixi run rs <d> check-deps --stale --mutex-only # only mutex violations
99+
pixi run rs <d> check-deps --stale --repodata https://conda.anaconda.org/<channel>
100+
pixi run rs <d> check-deps --stale --delete # delete stale local artifacts
101+
```
102+
103+
For published stale builds, the report prints the `pkg_additional_info.yaml`
104+
build-number snippet, a `mutex_package: build_number:` bump (re-publishes the mutex
105+
with new constraints without changing its version), and the `anaconda remove`
106+
commands for the old files.
107+
108+
## Bump vinca or rattler-build
109+
110+
Both are pinned once for all distributions in `pixi.toml` (vinca as a git revision).
111+
After a bump, run `pixi install`, `pixi run rs check`, and
112+
`pixi run rs <d> generate-recipes` for a distribution or two, and compare the recipe
113+
sets before and after.

0 commit comments

Comments
 (0)