stdenv: PURL fetcher introduction & feature flag

[h0nIg]

#421125 was merged and reverted later, because of regressions.

the background is described here: #421125 (comment)

@wolfgangwalther outlined the conditions and would like to enhance CI - over time. This is a continuous approach, which is in line with packages which have been found to be defunct and which need a fix. There may be more packages which have problems and we would like to prevent further fallout by a feature flag (prevents accessing + inheritance of drv.src / drv.srcs).

packages list: #453322 (comment)

With the old PR + the broken&platform check fix from #453291 + the feature flag, we enable maintainers to gather experience with PURL and set appropriate information (e.g. jq example, where fetchurl is used instead of fetchFromGithub)

nix-repl> xx = (import /my/nixpkgs {config={derivationPURLInheritance = true;};})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purl = "pkg:github/boto/[email protected]";
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

nix-repl> xx = (import /my/nixpkgs {})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[h0nIg]

we need to squash all commits later, just for transparency let's keep the commits separated in order to understand what has been changed.

[alyssais]

I'm still a bit uneasy about doing this without a plan for handling Cargo/NPM/etc libraries. I'd like to hear from other people who do think this is good to go (e.g. @arianvp) what they imagine the path to having complete pURL information to look like.

[raboof]

I'm still a bit uneasy about doing this without a plan for handling Cargo/NPM/etc libraries. I'd like to hear from other people who do think this is good to go (e.g. @arianvp) what they imagine the path to having complete pURL information to look like.

For my 2ct, I think being able to start recording/getting some pURL information is already helpful, even if it doesn't include cargo/npm/etc dependencies yet. For the use case of matching installed software to security advisories, the data quality on both sides is terrible, and being able to (more easily) also include pURL in the heuristics to relate those seems sensible. To go down the rabbithole a bit further: the fact that an advisory exists for a dependency doesn't necessarily mean the depending package uses that library in a vulnerable way: except for a few high-profile exceptions, I'd say more often than not, the deeper in the dependency tree the dependency with the advisory exists, the less likely it is that the system is actually affected by it in practice. For that reason, having better matching only at the 'shallow' side (while not yet having a solution for the 'deep' end) would be a good start. If formats like VEX gain traction, you could even imagine using the identifiers on the 'shallow' side to query for a VEX, for example in something like a TEA repo, and discover any relevant 'deeper' vulnerabilities via that route.

I agree it's all still rather speculative. I wish we had a way to mark structures as 'experimental', to make it easier to merge while signaling that we reserve the right to change things later.

[YorikSar]

I'm overall fine with this approach, but have a couple of suggestions below.

[alyssais]

Arnout Engelen ***@***.***> writes:

I agree it's all still rather speculative. I wish we had a way to mark structures as 'experimental', to make it easier to merge while signaling that we reserve the right to change things later.

Given that we /know/ we'll need to at some point identify dependencies we don't know about at eval time, though, it's not really an experiment, is it? We already know this can't be the ultimate solution, and that we'll instead need some sort of build-time recording of PURLs (perhaps to a dedicated output?).

[h0nIg]

Arnout Engelen @.***> writes:
I agree it's all still rather speculative. I wish we had a way to mark structures as 'experimental', to make it easier to merge while signaling that we reserve the right to change things later.
Given that we /know/ we'll need to at some point identify dependencies we don't know about at eval time, though, it's not really an experiment, is it? We already know this can't be the ultimate solution, and that we'll instead need some sort of build-time recording of PURLs (perhaps to a dedicated output?).

we do not need ecplicit build time dependencies, they are part of the SBOM / dependency graph. Identification is just required on individual drv's.

@alyssais did you tried an SBOM tool before (e.g. sbomnix, it has parameter to gather runtime only / runtime+buildtime)

edit: you meant FOD, got it

[h0nIg]

I'm overall fine with this approach, but have a couple of suggestions below.

@YorikSar kind reminder, your changes are incorporated and I think you need to approve as well. thx!

[yovko]

LGTM! I can't wait to use this!