rustdoc: add doc_link_canonical feature

[lolbinarycat]

proposing this as a t-rustdoc experiment (do we have a process for those?), implementation was surprisingly simple so it's basically a complete implementation of my design (although ideally it would be integrated into cargo in the future using --crate-attr), will likely need FCP then RFC for stabilization, may also need perf testing to make sure added allocations have a negligible impact, not sure.

fixes #143139

r? @GuillaumeGomez

Problem Statement

There are two main usecases for this feature:

1. crates that wish to have their canonical documentation site be somewhere other than docs.rs.
2. crates that want to have docs.rs be their canonical documentation site, but their crate docs are frequently included in doc bundles hosted by a third party.

In both these cases, this can cause search engines to duplicate results or show the wrong (non-canonical) page in results.

Solution

A new crate-level doc attribute is added, html_link_canonical, which adds a link rel="canonical" element to the head of every documentation page (pages such as help and settings are excluded due to not belonging to any crate in particular), leveraging the existing system search engines use for de-duplicating equivalent pages.

Future Work

1. When html_link_canonical is used with no value, it will use the value of html_root_url, or --extern-html-root-url, if present.
2. cargo flag that implies -Zrustdoc-map and passes --crate-attr='doc(html_link_canonical)' to each rustdoc invocation. This will cause all crates that do not manually specify html_root_url or html_link_canonical to use the docs.rs page as the canonical page. Alternatively, instead of implying -Zrustdoc-map, it could simply reuse its logic, passing the value that would be passed to --extern-html-root-url to be passed via --crate-attr='doc(html_link_canonical="...")'.

[rustbot]

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann

[lolbinarycat]

The one feature that isn't currently implemented is having the default value be the same as from html_root_url if you just specify doc(html_link_canonical) with no value.

[GuillaumeGomez]

Took a look at the implementation, it seems correct. However, I'm not sure it solves it the correct way, would need to actually completely understand the issue (don't have time right now but didn't want to leave the PR pending with no feedback).

So for experimental features, we generally go through an FCP. Now, to make a PR able to go through FCP, the first comment needs to contain a lot more information: describe exactly the problem it tries to solve, how it solved it and why it solved it this way.

[lolbinarycat]

@GuillaumeGomez did my best to describe the experiment, let me know if there's anything I can give more details on or write better.

[bors]

☔ The latest upstream changes (presumably #144181) made this pull request unmergeable. Please resolve the merge conflicts.

[GuillaumeGomez]

Thanks for the complete explanations. Time to start the fcp then. :)

@rfcbot fcp merge

[GuillaumeGomez]

Woops, too many teams pinged. Sorry for the notifications...

@rfcbot fcp cancel

[rfcbot]

@GuillaumeGomez proposal cancelled.

[GuillaumeGomez]

@rfcbot fcp merge

[rfcbot]

Team member @GuillaumeGomez has proposed to merge this. The next step is review by the rest of the tagged team members:

• @GuillaumeGomez
• @camelid
• @jsha
• @notriddle

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[notriddle]

crates that want to have docs.rs be their canonical documentation site, but their crate docs are frequently included in doc bundles hosted by a third party.

Based on what I've seen discussed in rust-lang/docs.rs#1438 I think there's a double-bind if you try to do that.

• you can't put a latest URL in here, because if this current crate is not the latest, it would result in a "canonical" link to a page that is different from the current crate, and that'll get you penalized
• you shouldn't put a versioned URL in here, because the docs.rs team wants the latest URL to get more link juice, so that users don't get outdated docs in their search results

So I don't think you can actually used it for that.

[lolbinarycat]

@notriddle couldn't you gate the attribute behind cfg(not(docsrs)) so that the attribute is only generated for third-party docs?

ofc, that would either have to wait for stabilization, or it would require a seperate nightly feature, so it wouldn't be a perfect solution.

[notriddle]

couldn't you gate the attribute behind cfg(not(docsrs)) so that the attribute is only generated for third-party docs?

The problem isn't with linking from docs.rs. The problem is linking to docs.rs. There's no URL to point at that simultaneously satisfies the rel="canonical" law and the team's SEO goals.

Outside of docs.rs, you've still got the same core problem: you have to make sure that the page you link to has the same crate version, not just the same name, or Google will ignore the link. Other than docs.rs and doc.rust-lang.org, most sites have no way to do that, because they don't have version numbers in their URLs.