Mark span parent in def_collector.

[cjgillot]

The current device of marking spans with a parent def-id during lowering has been frustrating me for quite some time, as it's very easy to forget marking some spans.

This PR moves such marking to the def_collector, which is responsible for creating def-ids on partially expanded AST. This is much more robust as long as visitors are exhaustive.

r? ghost

[cjgillot]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 95dc7f7 with merge ec84e1c...

[cjgillot]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 585fe45 with merge 04122fb...

[bors]

☀️ Try build successful - checks-actions
Build commit: 04122fb (04122fbfda6e59dcf9c54b89353e6d1281c9ebc8)

[rust-timer]

Finished benchmarking commit (04122fb): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.0%	[0.2%, 2.5%]	163
Regressions ❌ (secondary)	1.3%	[0.3%, 6.2%]	77
Improvements ✅ (primary)	-2.6%	[-4.8%, -0.3%]	12
Improvements ✅ (secondary)	-0.5%	[-0.7%, -0.4%]	2
All ❌✅ (primary)	0.7%	[-4.8%, 2.5%]	175

Max RSS (memory usage)

Results (primary 0.3%, secondary -0.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.1%	[1.4%, 2.6%]	6
Regressions ❌ (secondary)	4.2%	[4.0%, 4.6%]	3
Improvements ✅ (primary)	-2.4%	[-4.4%, -1.0%]	4
Improvements ✅ (secondary)	-3.6%	[-4.3%, -3.2%]	5
All ❌✅ (primary)	0.3%	[-4.4%, 2.6%]	10

Cycles

Results (primary 0.6%, secondary 3.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[0.7%, 3.4%]	28
Regressions ❌ (secondary)	3.2%	[2.1%, 4.3%]	11
Improvements ✅ (primary)	-4.9%	[-6.0%, -1.4%]	7
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.6%	[-6.0%, 3.4%]	35

Binary size

Results (primary -0.2%, secondary -0.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.2%]	7
Regressions ❌ (secondary)	0.1%	[0.1%, 0.2%]	6
Improvements ✅ (primary)	-0.2%	[-1.7%, -0.0%]	41
Improvements ✅ (secondary)	-0.3%	[-0.9%, -0.0%]	13
All ❌✅ (primary)	-0.2%	[-1.7%, 0.2%]	48

Bootstrap: 695.108s -> 697.834s (0.39%)
Artifact size: 327.55 MiB -> 327.54 MiB (-0.00%)

[cjgillot]

It would appear that half the regression in incr-unchecked comes from Span::to in this line. The other half comes from using a MutVisitor in DefCollector, which had to be expected.

[petrochenkov]

The other half comes from using a MutVisitor in DefCollector

I tried to address this in #127371, not sure if it will help.

[cjgillot]

@petrochenkov do I need to add something to this PR to handle metavar spans, or is this pass enough?

[cjgillot]

@bors try @rust-timer queue

[cjgillot]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 2789588 with merge 327a748...

[bors]

☀️ Try build successful - checks-actions
Build commit: 327a748 (327a748314e67e61efda8c2ef79ebf268d6ec4cf)

[rust-timer]

Finished benchmarking commit (327a748): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.1%, 2.6%]	124
Regressions ❌ (secondary)	0.8%	[0.1%, 2.6%]	73
Improvements ✅ (primary)	-0.6%	[-1.1%, -0.2%]	12
Improvements ✅ (secondary)	-0.4%	[-1.3%, -0.1%]	37
All ❌✅ (primary)	0.7%	[-1.1%, 2.6%]	136

Max RSS (memory usage)

Results (primary 1.6%, secondary -2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[0.5%, 6.8%]	5
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.9%	[-0.9%, -0.9%]	1
Improvements ✅ (secondary)	-2.0%	[-4.2%, -1.1%]	5
All ❌✅ (primary)	1.6%	[-0.9%, 6.8%]	6

Cycles

Results (primary 2.9%, secondary 3.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.9%	[1.5%, 4.2%]	14
Regressions ❌ (secondary)	3.8%	[2.9%, 5.0%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.9%	[1.5%, 4.2%]	14

Binary size

Results (primary -0.1%, secondary -0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	4
Improvements ✅ (primary)	-0.1%	[-0.2%, -0.0%]	44
Improvements ✅ (secondary)	-0.2%	[-0.5%, -0.0%]	18
All ❌✅ (primary)	-0.1%	[-0.2%, -0.0%]	44

Bootstrap: 462.57s -> 465.918s (0.72%)
Artifact size: 374.65 MiB -> 374.70 MiB (0.01%)

[bors]

☔ The latest upstream changes (presumably #144145) made this pull request unmergeable. Please resolve the merge conflicts.

[cjgillot]

I wonder if it would be possible to assign span parents even earlier, during parsing.

That way we'd automatically "visit" everything including tokens in macros, and also avoided mutating AST (if immutable arena-allocated AST is a goal).

We'd keep a current parent id in the parser state and update it when recusing into things like items. The parent id doesn't even need to be a DefId or match the DefId hierarchy exactly, it just needs to be close enough to its inner spans to resist small changes and we can map these ids to DefIds later (e.g. in def collector).

Upd: if we do this and remove nonterminals (AST pieces inside tokens), then we'd be able to remove MutVisitor::visit_span as well (it will only be needed for token streams, but not AST).

I'm only getting your point now. As the parser does not have DefIds yet, we'll need to find another way. I see 2 possibilities:

1. create an ad-hoc hierarchy SourceItemId, with some device like DefPath for stability, and use them as parents;
2. create a LocalExpnId for each item and use them for this.

Solution 1 seems like reimplementing a lot of code. For now, expansions are probably too unstable due to hashing spans.

What did you have in mind?