POC of using cargo package for installation

[luca-della-vedova]

This PR targets #67 and illustrates the idea I have been toying with about using cargo package to install libraries through colcon-cargo.
It's just a POC to facilitate discussion, ideally we can at least be smarter with caching / remove duplicated work and rework install folder structures.
The proposed implementation in #67 uses cargo package --list to get a list of files that would be installed then uses the output to copy files from the package's source directory into the install space.
This however, is not sufficient for packages that are in a cargo workspace, since cargo package actually performs a series of operations to the Cargo.toml to, for example, remove relative path dependencies, resolve workspace dependencies etc. (point 1 and some of point 2 here).
This shows in the fact that what we actually install in #67 doesn't compile if applied to a more complex package.

I came up with a "simple" test case for both #67 and this PR to show the difference.

Setup

Create a workspace and bring a custom tokio (where I just added a public hello_world function) and a POC crate that tests it (has a library and an executable that call the new tokio function).

mkdir -p tokio_ws/src
cd tokio_ws/src
git clone https://github.com/luca-della-vedova/tokio
git clone https://github.com/luca-della-vedova/cargo_custom_tokio_package

Test with #67

Remove dev_depends from here as done in this PR to avoid circular dependency errors

# TODO Install or checkout the colcon-cargo branch in #67
cd ~/tokio_ws
rm -rf build install log
colcon build --packages-up-to tokio
cd install/tokio/share/cargo/registry/tokio-1.48.0
cargo build

Since the tokio Cargo.toml has one of these "workspace inheritances", the build will actually fail, making the crate effectively non functional:

:~/tokio_ws/install/tokio/share/cargo/registry/tokio-1.48.0$ cargo build
error: failed to parse manifest at `~/tokio_ws/install/tokio/share/cargo/registry/tokio-1.48.0/Cargo.toml`

Caused by:
  error inheriting `lints` from workspace root manifest's `workspace.lints`

Caused by:
  failed to find a workspace root

The snippet is preserved when copying the file and cannot be resolved since the file is copied to a different location and its workspace's Cargo.toml is not in its parent folder anymore.

[lints]
workspace = true

Test with this branch

# TODO Install this PR's colcon-cargo version
cd ~/tokio_ws
rm -rf build install log
colcon build --packages-up-to tokio
cd install/tokio/share/tokio/rust/tokio-1.48.0 # I just put it here to follow what we do with other rust packages, happy to bikeshed
cargo build

You will see that the package builds correctly, meaning the workspace migration was done by Cargo.
Inspecting the installed Cargo.toml, we can see that the linter configuration was migrated to a standalone configuration that does not reference the workspace's Cargo.toml:

[lints.rust.unexpected_cfgs]
level = "warn"
priority = 0
check-cfg = [
    "cfg(fuzzing)",
    "cfg(loom)",
    "cfg(mio_unsupported_force_poll_poll)",
    "cfg(tokio_allow_from_blocking_fd)",
    "cfg(tokio_internal_mt_counters)",
    "cfg(tokio_no_parking_lot)",
    "cfg(tokio_no_tuning_tests)",
    "cfg(tokio_unstable)",
    'cfg(target_os, values("cygwin"))',
]

Additionally, if now you build the sample package:

# TODO Install this PR's colcon-cargo version
cd ~/tokio_ws
rm -rf build install log
colcon build --packages-up-to cargo_custom_tokio_package

It will build successfully and running it will work, meaning the hardcoded patch correctly resolved to a tokio with the custom function:

$ install/cargo_custom_tokio_package/bin/cargo_custom_tokio_package 
HELLO THERE

Conclusion

• Just copying files is not sufficient, cargo package does a lot more for non-toy projects. I demonstrated this with tokio but most major projects out there have workspace dependencies that would make them non functional (i.e. serde).
• This approach should make packages functional and use a similar approach to what we currently do in colcon-ros-cargo / message generation. Patching is hardcoded but left as future work (perhaps through pallet patcher?)
• There are rough edges, specifically I wonder if cargo package always builds a package even if it was unchanged, since I see that install times are quite significant even when no changes are made.

[Blast545]

I didn't test the example, but checked the code and the documentation of cargo package and this LGTM

[Blast545]

Do you know why this causes the circular dep problem?

[luca-della-vedova]

It's an interesting issue.
Cargo itself actually allows circular dependencies for dev dependencies but colcon doesn't, so if we add all dev-dependencies to our build depends, we might fall into some false positives.
Details on when and why is this allowed are in the cargo book. From my understanding it's because a test is a standalone target that is built separately from the library so there isn't strictly a circular dependency.
The documentation advises against this practice but, of course, a lot of packages in the wild go and use it, such as tokio or serde that are very widespread.
Note that however this is just a hack to be able to test this PR, I don't think we should merge this change. I believe what we would really want is indeed include dev dependencies in our dependency tree, but at the same time disregard dependency cycle errors only when they occur while resolving dev dependencies (and perhaps only cargo dev dependencies). This is however quite tricky and we should probably explore it separately.