Model extras as a single resolver node

[notatallshaw]

I went down a rabbit hole looking at how extras were implemented in pip from the following issue: #14139

Resolve a project and its extras as one node instead of a separate node per extra. identify() returns the bare project name, so foo and foo[extra] share a node resolved once against the merged set of requested extras. This removes the per-extra nodes and the synthetic base-dependency edges that kept their versions aligned.

In my own benchmarking suite, apache-airflow[all] drops from 11,234 to 2,774 requirements visited and from 740 to 612 resolution rounds, and a warm-cache pip install --dry-run runs about 8% faster. I found no regressions, although a small number of solutions change due to the resolver visiting some extras in a different order.

The careful part of this work was that a node stays pinned while any requirement on it holds, so it can keep an extra whose only requester was backtracked away. This requires a pass to rebuild the install set under only the surviving extras and drop that extra's dependencies (the per-extra model got this for free). When nothing is stale it reuses the edges resolvelib already recorded, so the performance cost is de minimis.

Finally, this moves marker evaluation onto the requirement node and causes a dependency with ; extra != "x" to no longer be pulled in when x is requested. This does not help solve this problem for transitive resolutions though; see the discussion in https://discuss.python.org/t/ban-negative-extras-for-extras-marker/108022

[pfmoore]

Wow, awesome work! This was always the most annoying part of the new resolver code - I know I tried a few times to get rid of the weird extra node, but couldn't make it work.

I don't have time right now to do a review, but I look forward to seeing how you got this to work 🙂