RFC: Extended Standard Library (ESL)

[bdaehlie]

This RFC describes an Extended Standard Library (ESL) for Rust.

Please see this document for additional information about fundraising.

I’d like to thank Dirkjan Ochtman (@djc), Adolfo Ochagavía (@aochagavia), Alex Gaynor (@alex), James Munns (OneVariable GmbH, @jamesmunns), Andrew Gallant (@BurntSushi), Marco Ieni (@marcoieni), and many others for their feedback and assistance while working on this. My thanks does not necessarily imply their endorsement.

Rendered

[Manishearth]

implementation effort is ongoing. I don't think this ideal is difficult at all.

I'd argue that rustlang::foobar (or rust_esl::foobar) for ESL crates is a pretty good use of that feature, and is a good place to put funding if things need to be expedited.

I don't see another easy way to have this be clear from the names in a way that's not easily spoofed.

[bdaehlie]

If we could pull it off that would be ideal. It's clearly better. But it depends on namespacing being implemented and having the relevant maintainers on board with renames.

[Manishearth]

It seems like renames are necessary here anyway? Maybe I missed something but I think a very important facet that isn't being really discussed in depth this RFC is how ESL membership is signified.

[bdaehlie]

Right now the RFC does not require renaming crates, only moving them to the ESL GitHub organization:

"As such, pre-existing crates that join the ESL will keep their names unless there is a compelling reason to make a change" ... "Crate users will have to confirm ESL membership by checking the GitHub organization."

The ability to confirm membership is important. Originally my draft RFC did require RFC 3243 implementation and renames into an ESL namespace in order to provide clarity about functionality and membership. Concerns were raised about the difficulty of renaming existing popular crates, so this compromise was made. If the community thought renaming crates into an ESL namespace was feasible that would be a better outcome and I'd be happy to update the RFC as such. I'd want to hear from potential ESL crate maintainers first though.

[bdaehlie]

Also, I don't know if you got there yet, but the Naming and Namespaces section of "Future Possibilities" further discusses crate naming and indicating membership.

[Manishearth]

The ability to confirm membership is important.

I think this RFC would be more compelling if it spent significant time talking about this: what the options are, what the "minimum viable" thing is for this RFC to be considered implemented.

In its current status "make a github organization" doesn't really feel like a compelling RFC for the Rust project. I understand that there are no readymade solutions on the table available right now, but that doesn't mean you can't talk about what the solutions could be. If this RFC needs additional cargo features to make sense, so be it.

It seems like the primary goal of this RFC is trust, and it spends a surprisingly small amount of time talking about mechanisms for achieving trust. It talks about what makes up a trustworthy crate (good!!), but being trustworthy does not make something trusted: how do you plan to make it easy for developers to trust these crates? Goal #2 explicitly talks about there being a subpar experience on choosing crates, but I don't see how this really helps that developer experience along.

Concerns were raised about the difficulty of renaming existing popular crates

"Renames as redirects" is a thing that could be proposed. Indeed, we were considering proposing it as a part of the namespaces project before deciding to defer that for later.

[bdaehlie]

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

The concern pointed out to me about renames was not so much about literally renaming or redirecting crates, but about the volume of extant documentation, both formal and information (e.g. blog posts, stack overflow, that sort of stuff).

[epage]

This is organizational namespacing which goes against the spirit of that RFC.

I was going to raise a separate topic that naming should be out of scope.

[Manishearth]

@epage Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

My point here is less about reintroducing the namespace requirement and more about bringing more focus onto how we make this trusted relationship clear to users. There are many mechanisms to do so, namespacing is one of them.

[epage]

Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

There are trade offs to using this outside of its intended purpose. Our example of doing so will be noticed much more than any caveats we put in the documentation around this feature.

[ChrisDenton]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

[JohnDowson]

Some, perhaps, even a language feature. Bitfields probably being the least controversial candidate.

[ssokolow]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

As someone who's been here since about 2013, it's been my observation that Tranche 1 stuff that isn't in the standard library had very good reason to not be in the standard library back then, and the question is whether they've managed to stabilize their APIs enough in the intervening 12 years for that to no longer be an issue.

I can literally name, off the top of my head, at least one (sometimes two or more) formerly-leading implementations of every item on the latter half of that list which were, at the time, the leading solution:

• Random number generation (rand has gone through multiple API breaks)
• Serialization/deserialization (Serde supplanted rustc_serialize, which was once the leading solution)
• Data structure serialization (Same thing. We'd have been stuck with rustc_serialize if we moved earlier.)
• Date/time support (We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.)
  Error handling support (error-chain to failure, failure to anyhow and thiserror, with eyre around 1/7th of anyhow's popularity and climbing... and that doesn't count the ecosystem's "Yeah, no." reaction to a team member's idea of what a good error-handling API should look like in fehler.)

It may be slow, but things do make it into std in good time, as demonstrated by the incorporation of a variation on once-cell as it became apparent that it was both the superior API and slowly but surely winning out over lazy-static.

...but, at the same time, despite the team's best efforts, things like the two deprecated methods on the Error trait do still exist.

I will admit that I haven't noticed much churn in the top half of the list, but those are also niches I don't pay as much attention to.

[Sky2042]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

[ssokolow]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

Ahh, yeah. That sounds/looks right.

I tend to have burntsushi stuff that I'm not already in the habit of using slip my mind because using crates.io is just so much less comfortable than using lib.rs.

[epage]

I think how the ESL would evaluate time vs chrono and then deal with the creation of jiff would be a good test case for this. There was a lot of discussion on blessed on just how to talk about these 3.

Error handling support (error-chain to failure, failure to anyhow and thiserror, with eyre around 1/7th of anyhow's popularity and climbing... and that doesn't count the ecosystem's "Yeah, no." reaction to a team member's idea of what a good error-handling API should look like in fehler.)

failure, anyhow, and eyre all follow along the same design path and I think its local maxima that is just good enough that experiments in other directions are not getting enough attention. Blessing eyre, in std or ESL, would likely make this worse.

[joshka]

I think how the ESL would evaluate time vs chrono and then deal with the creation of jiff would be a good test case for this. There was a lot of discussion on blessed on just how to talk about these 3.

nicoburns/blessed-rs#151 for anyone looking for that conversation.

[Manishearth]

Personally I think the time crates are an example of something that shouldn't ESL for the reason that they solve a hard problem where API design actually really affects things, so I would expect a lot of different approaches, all of them being valid.

If we're worried about picking winners (and we should be), the stuff we should pick is where multiple different implementations can be expected to differ on, at most, internal implementation quality.

the regex crate is a good example of this. there's not too much interesting about a regex API, there are some different ways of doing it but the tradeoffs are minor. (That said, lack of features like backreferences is the kind of thing that is technically API and can differ. Seems like by and large the rust community doesn't seem to want those features anyway)

Time handling is something where we've seen a lot of innovation within the rust community and without, because it's a hard subject. I'd be very wary of picking winners there. Even if the current crate maintainers were fine with it, it could potentially cause strife in the future.

I have a separate thread on this below, but basically I think significant effort needs to be put into deciding when it is ok to pick a winner in the first place, before we figure out how to pick a winner. My suspicion is that most crates where picking a winner is tricky are ones where we probably shouldn't be doing so.

[SOF3]

Is there still a real motivation of moving these to std at the cost of future compatibility burden beside concerns over supply chain reliability, which is precisely the point of this RFC?

[Lokathor]

Do you have any numbers or other indications to demonstrate that the reliability of software has been hurt by the status quo? I simply don't accept the basic claim that decentralized code is bad and that we can solve things via centralizing. If anything, the rust project is one giant indication that centralization is a quick way to bury a small amount of people under a massive amount of work, until it all slows to a crawl because too many things are looked at by too few eyes.

[JohnDowson]

But is this RFC about centralisation of development, or is it about

1. ) giving users a seal of quality for some third party crates
2. ) providing funding to the maintainers of such approved crates

To me it reads like it's more of the latter.

[Lokathor]

The entire thing is about moving approved crates into an approved github org so that only approved people can access any of it. The goal is a stamp of approval, the mechanism is centralization.

[bdaehlie]

The maintainers of the crates that move into the ESL would continue to be the maintainers. Anyone could contribute to the crates with PRs, same as before.

The GitHub org would enforce some best practices, particularly with regards to security, and additional resources would be available to help with maintenance. I think this is important for the ecosystem's most commonly used crates. Having them differ in terms of compliance with best practices, security policy and resources, and implementation of infrastructure and controls across many different GitHub orgs is not a good thing for Rust security, and it's probably more work in aggregate.

[Lokathor]

So everything is the same, except when it's not the same? That seems incoherent.

I don't ask all this totally out of nothing. tinyvec is a crate that's available in the rust playground. I take this fact as a burden to be extremely careful of. How, concretely, does me moving my repo from my GitHub acocunt to your theoretical org meaningfully change anything for the better? Can you point to actual security situations that have occurred with rust that would have been solved if this RFC had been implemented before the event?

[the8472]

I meant this one: rust-lang/crates.io#11131
It's basically outsourcing the process management of creating a trusted release to github (and other trusted publishing platforms). Then the org can limit publishing to bots instead of humans and the bots will only publish after the appropriate reviews.

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

[epage]

Slippery slopes...

"Trusted Publishing"s name isn't meant to say other forms of publishing aren't trusted but to secure specifically focusing on the CI publishing process. The misleading name already caught on though.

There are trade offs with the current state of bot-based publishing such that we should not require only that. Without requiring bot-based publishing, you then need to allow direct pushes unless you make the release process onerous.

[matthieu-m]

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

I personally would wish for quorums, here.

Automate the publication to crates.io, certainly, but then leave the crate in quarantine mode until other maintainers or dedicated auditors have vetted the published version.

Even small quorums (1 single independent auditor) would massively raise the bar compared to the statu quo.

(On projects where a single maintainer could also trigger a release by a bot, which is likely, you'd want a minimum quorum of 2, obviously)

[the8472]

A) a trusted publishing process may already have a quorum on the CI side (github deployment environments can require multiple approvers), so reimplementing the same functionality on cratesio seems redundant.

B) If we're involving independent authors this seems like out of the hands of the maintainers, in that case "quarantine" would be the wrong framing imo. I think users being able to filter crates on cargo add and cargo update by whatever security policies they want to apply would be more appropriate than making it a global decision.
AIUI big orgs already run their own crate mirrors where they filter which crates can be pulled.

[epage]

Depending on how its done, quarantining packages could cause major problems for workspace maintainers. It would help if we had a whole new publish API but that has been talked about for years. Even once you have that, there can be reason to publish a workspace in fragments.

[clarfonthey]

I don't think that this RFC is a good idea for a couple reasons.

The first is, quite frankly, I don't believe that this project will actually improve the situation, but in fact, make it worse. I do not believe that the Rust Foundation will be able to secure more funding to develop the necessary crates than these crates will ultimately be able to get on their own, which I'll get into later. I also believe that by "spreading thin," changes to these crates will ultimately be reviewed less often and lest thoroughly, meaning that it only increases the risk of "supply chain" issues it purports to prevent.

The second is that, flat out, this does not seem to understand the primary reasons why "supply chain" attacks happen, and thus will fail catastrophically to prevent them. The "secret sauce" involves some combination of the below ingredients:

• Widespread use
• High demand for changes
• A small number of maintainers
• A large amount of work per maintainer

Rust mitigates some of these issues, but not really. While unsafe code is heavily scrutinised, a large portion of the Rust ecosystem still relies heavily on external code without such promises. Even the Rust compiler itself, despite strongly working toward "oxidising" its internal dependencies (Rustup recently moved to being RusTLS only, for example), it still relies on massive unsafe code bases like LLVM and Git. While the number of these will decrease over time, I don't expect this to ever fully go away; even if all of the code Rust itself uses is written in Rust, it can still call down to the OS which may not be. And this doesn't even begin to approach exploits which don't even touch unsafe code.

It also particularly points out the XZ vulnerability, which I think is a perfect example that could happen in the Rust ecosystem right now, given the right circumstances. Right now, we have a small number of maintainers on each team which are expected to sign off on a very large number of changes created by anyone, and the Rust foundation has approximately 12 employees at the time of writing, with fewer people actually managing changes to the code itself.

But don't worry! The power of Rust is that many corporations find it in their best interest to hire people full-time as maintainers too!

Do you trust Microsoft, or Amazon, or Google to always hold the community's best interests at heart and not once, never, ever potentially exploit it? Do you think that this does not put them in an excellent position to leverage their community trust for exploiting any number of things, ever, in perpetuity? Do you trust every person in a position like this to properly back out and tell the project what's going on instead of exploiting their position like their employer wants?

Ultimately, there isn't a great solution for "supply chain" attacks that the tech industry wants, because its misdoings are the reason why such attacks exist in the first place. Moving crates into central control and telling people that the foundation has solved the supply chain problem, however, is actively harmful, because it hasn't solved the problem and can't.

I also would love to see more of these crates become supported and maintained by the entire community, but that's not going to happen by just waving a bureaucratic wand. And I think that pretending that the problem is solved will only make Rust a ripe target for these attacks.

[hawkw]

The drawbacks elaborated in this section are likely to be significant challenges facing such an effort. I feel the current discussion of these difficulties furnished here doesn't sufficiently discuss how these challenges will be handled. In particular, I have to say that simply stating "we will make the difficult decisions" and "we will reach consensus while keeping analysis paralysis at bay" does not inspire confidence in the ESL effort. These are serious issues, and the ways in which they will be addressed is, in my opinion, the most important part of a serious proposal to establish something like the ESL. Similar issues of community conflict and opaque decisionmaking processes have caused substantial problems for the Rust community in the past, and a proposal to establish "blessed" crates that doesn't anticipate such issues and propose a detailed plan for how they will be addressed seems likely to cause more conflict than it resolves.

I think these issues deserve substantially more discussion and forethought than is currently presented in this RFC. They cannot just be handwaved away by saying "we will make the right decisions".

[bdaehlie]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

[Manishearth]

It's worth noting that previous such attempts for a blessed "extended stdlib" died on similar rocks. We have had tremendous conflict in the past when it comes to Rust "picking winners", even with situations where there's just the perception of Rust picking a winner. Folks have quit the community due to this type of thing.

There's some number of libraries where Rust chose to tip the scales (libc, regex, etc) by making them Official Libs Team projects, and that's mostly cases where the ecosystem mostly settled on a very clear "winner" (like regex), or where it's the type of thing that really should be managed by Rust (libc). Arguably, syn could fall in that category.

There's also a worry of interop: even if you bless regex as a winner, codebases using regex can easily interoperate with codebases using other regex engines because it's just a library you call. This is different from, say, picking a serialization library or async runtime, where the decision can be much more viral.

(syn is an interesting example because it's a massive API surface which tends to be a problem for ecosystem interop, but on the other hand 99% of syn users use it as an internal dependency, so it ends up mostly being okay. mostly)

I think it's possible to come up with a set of principles for the types of crates where it is acceptable to pick winners. I think it'll take significant effort, but it's doable. I think that to be a prerequisite for such an RFC; otherwise we risk significant drama and hurt feelings in the future with bad decisions.

At the end of the day, making a decision is important even if it is not "perfect" by a given set of standards.

I strongly disagree with this. It is not important to uplift a library for any given function. It's a nice thing to have. The downsides here are tremendous, and their associated costs are measured in people.

[hawkw]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

If the intent is to use existing decisionmaking processes within the Rust foundation here, I think it would be worthwhile to explicitly state that, at the very least. Ideally there would be some discussion of how these processes apply to the decisions necessary to establish the ESL.

Fundamentally though, I agree strongly with the points in @Manishearth's comment. What you are proposing is something that has the potential to cause a great deal of conflict in the community, and previous efforts to do similar things have failed largely due to that conflict. How are you going to solve those problems this time?

If the goal is to allow users to confidently depend on the crates in the ESL, then it is necessary to establish community confidence in the process through which those crates are selected and through which technical decisions are made in the course of their maintenance. What you want to do here requires getting both maintainers and users of crates on board with the proposal. I think that the right time to sow the seeds of such a sense of confidence is now. Failure to convince the community that the processes used to make these decisions are fair and robust means that the best-case outcome for the ESL is a future where most users prefer to continue depending on independently-maintained crates due to higher levels of confidence in their decision-making processes than those of the crates in the ESL.

[Manishearth]

If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

Those processes have historically been unable to successfully make difficult decisions around elevating ecosystem projects. Really, anything involving people and feelings over code is super tricky and is not what the Rust decisionmaking process is designed for.

It's understandably not obvious from the history, but I'd say figuring this out is the core prerequisite for any such proposal.

[hawkw]

For the record, I want to make it clear that I am not expressing opposition to the idea of having something like the ESL. Instead, I'm pointing out that there is a reason that there are so many previous attempts to do something like this — if it had worked in the past, we wouldn't have to be proposing it now. If this attempt is to succeed, it must address the issues that have led to the failures of past attempts. Perhaps it will do so successfully, but the text of this RFC doesn't give me a reason to believe that.

[the8472]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

[epage]

Async, cli parsers, and many other cases involve trade offs and selecting which trade offs to bless will be difficult.

Some ideas

• Bless more than one, like blessed.rs sometimes does
• Bless the most minimal one (std-like)
• Bless the most flexible (full batteries)

[ssokolow]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

Honestly, I think it's premature to bless any async runtime before the "tell a complete async story" process is far enough along to start abstracting the APIs that tie projects to specific runtimes and adding those to the standard library as traits.

It wasn't that long ago that we got the "async fn in trait" MVP.

[the8472]

We have git as a tool to share code. We have github as a community platform to make the code-sharing and communication around it public.
We have cargo vet (and crev) as tools. What we're lacking is a visible platform for aggregating reviews, the social dynamics around 3rd-party code reviews. It's not a publishable good.

So I agree it doesn't go far enough. But I think rather than centralizing a few critical pieces a more scalable solution would be to record the figurative eyeballs that went over all the code, and the lack of eyeballs too.

Netflix, Huawei and the sovereign tech fund assigned engineers to review this code is useful information.
10000 people use this library but nobody ever reviewed it is useful information too.

If everyone (and every company) reviews in silence we repeat efforts and don't know who else already did the thing. It feels very much like a coordination problem. Nobody reviews a thing because reviewing everything is too much work.

This could start with surfacing such information on crates.io.
And "review crate of the day" kinds of efforts, make it a vscode plugin, and...

Otherwise I think the issue will be that any hypothetical supply-chain attacker will move their focus to "widespread use, but just outside the ESL scope" crates, which will remain unreviewed...

[epage]

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

There is that and other tacticts which we should work towards to improve supply chain management. I worry this might distract us from that larger problem because this can't cover everyone's needs but people will likely be optimistic about this effort

[Manishearth]

What we're lacking is a visible platform for aggregating reviews

n.b. libs.rs does show vet. as does deps.dev.

My comment below suggests a way we could make great use of cargo vet for attaining the goals in this RFC.

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

Yes, 100%.

[Manishearth]

I think this might be selling cargo vet short. The design is not for one to trust random audits; it was thus designed precisely because trust is multidimensional and contextual. It is a framework upon which to build a trust model. cargo-vet doesn't care about a project's security policies because cargo vet shouldn't be making that decision: instead, it lets one define their own criteria for what constitutes "good security policies" and filter based on that.

A very feasible way of achieving many (not all) of the goals of this RFC would be to define a set of criteria, and have some blessed audit repository for those criteria¹. It could be automated such that crates whose maintainers are trusted to follow these criteria automatically get audits published alongside their crate publishes. Sites like lib.rs could be petitioned to elevate such audits with a badge.

"Trust the entity that produced the code in the first place" is definitely a possible thing to represent within cargo-vet's framework.

(Potentially crates.io as well, though that's the threshold where you would need an RFC, up to this point this can be done entirely as a community effort).

Footnotes

1. audits retain their source: it is possible to distinguish a Google audit from a Mozilla audit from a rustlang audit. ↩

[epage]

This should be out of scope. Renames are a breaking change and requiring one to add/remove branding is a bad idea.

[Manishearth]

In theory cargo/crates.io could support "redirect" renames. It's a thing that was briefly discussed during the namespacing RFC, it didn't end up happening, but it's ... something that could be proposed.

I agree that without such a change it's probably not going to work.

[epage]

We should not count on being able to do so though as I expect some tricky problems in the resolver and Cargo's ideals around ‘Cargo.lock` files.

[epage]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

[epage]

(I have similar concerns about several others of these ideals)

[the8472]

I would even go a bit further, I think calling these requirements to be trustworthy is corporate-biased and mildly offensive to individual maintainers.
Saying that you're not trustworthy because you can't guarantee to maintain something well into the future or because you don't want to deal with the security stress and just want soundness bugs treated like any other bug doesn't make a dependency un-trustworthy. It just might make it fall below the level of unpaid support that some corporations might want.

But well

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

To me trustworthiness in something that's provided for free only means non-malicious and not having a track-record of bad decisions. Everything else sounds more like paid support or nice-to-haves.

Some things don't require eternal vigilance, e.g. hex hasn't seen a release in 4 years and imo this doesn't impinge on its trustworthiness, it might eventually become a problem for its usefulness.

[jieyouxu]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

To add to this, even the compiler, standard library, and various rust-lang/* crates like cc or libc have limited reviewer bandwidth, especially in specialized areas of the codebases.

• Has sufficient maintainer bandwidth to publish security updates in a reasonable timeframe
• Has reasonable code review and testing standards

Are really, really hard problems. I fully echo the

assuming more than can be expected or will give people a false sense of expectations for the result

concern.

[jswrenn]

How would removal work in practice? If the name of the crate is what conveys its membership in the ESL, what happens when that crate is removed from the ESL? Are its versions yanked?

[jswrenn]

How will the ESL ensure that existing maintainers are non-malicious? How will the ESL ensure that new maintainers of ESL crates are non-malicious?

[frewsxcv]

And similarly, who decides whom is qualified?

[BurntSushi]

I feel like this is largely a non-issue. Who decides who is on libs? Or the lang team? How do we know everyone on the teams is not malicious? How do we know they are qualified?

[frewsxcv]

If it's not important and up for debate, then what is this line for?

[BurntSushi]

It seems like it's elaborating on an important goal to me.

[epage]

This is a real tension where I have to decide between

• Being blessed or fading into obscurity
• The policies if they are too heavy

[jswrenn]

The drawback of picking winners isn't that folks "could be upset", it's the (good) reasons for which they would be made upset. If you maintain a crate whose competitor is selected for inclusion in the ESL, there will begin to be tremendous pressure to avoid your crate. If you have an idea for an alternative approach to a problem "solved" by the ESL, there will be tremendous pressure to avoid your crate. The ESL has the potential to stifle innovation in the crate ecosystem.

Had the ESL existed in 2018, I don't think I'd be paying my bills by maintaining one of the crates I maintain today; there simply would have been too much pressure to use the then-dominant alternative.

[BurntSushi]

I somewhat feel like this is already true today. I think the fundamental difference between the status quo and this proposal is that this proposal has a (presumably small) group of people making a determination about the winner as compared today where a winner tends to manifest via inertia or other pressures. But we certainly don't need an ESL to have the problem of it being almost impossible to unseat an existing dominant solution.

[Lokathor]

But I think we can definitely avoid making it worse at the least

[hawkw]

There's also the issue that, in many cases, there may be multiple crates for a given purpose which make very different design decisions. Often, these design decisions are equally "correct", and the right choice depends on the particular use case. In a situation like this where two or more crates that have a comparable level of maintenance quality make substantially different and incompatible design choices, what would the ESL do? Would both alternatives be added to the ESL? Or would the ESL just choose one such crate and leave users whose particular use case prefers the other design without an ESL-blessed option?

[fintelia]

Also worth calling out that choosing only one competitor means that the other ESL crates are forced to make the same pick. Basically...

We'd love for your crate to be part of ESL, but of course you'd have to switch from fastrand to rand. I'm sure your users won't mind!

[hawkw]

Here's a fun question that it would be nice to have an answer to: what happens if an ESL crate depends on a non-ESL crate? Now, you have a crate that's part of the trusted base of vetted libraries which, nonetheless, has the potential to "get xz'd" by its non-ESL dependency. Would ESL crates be forbidden from depending on non-ESL code? Would adopting a crate into the ESL require ESLing its entire dependency tree?

[epage]

I worry this will be too restrictive. I'm splitting up toml. If it was in the ESL, I'd have to get those lower level crates to be blessed to a similar level which also unduely raises their visibility withinea blessed list.

I worry this can also be stagnating with other dependencies.

[the8472]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.
https://forge.rust-lang.org/policies/crate-ownership.html

[epage]

Another example: I'm working on a replacement for libtest. It has its own cli parser and soon a json writer to meet very specific design requirements. Abandoning those requirements would be rough blessing each part would also.

In both examples, some of the packages are also being developed on an experimental basis. And the libtest project just left rust-lang org because of how strict the requirements are, even for experiments.

[Lokathor]

It's unfortunately demonstrably true that, until we have multi-threaded rustc, it's faster for builds to split crates into multiple sub-crates as often as possible. We probably want to encourage large crates used by many people to break up their code as much as possible, rather than encouraging single large crates.

[epage]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.

The Project has that but this is separate. That policy is also very strict in that something can only be Intentionally provided by the Project or its solely Internal to the project. Some of the cases I'm referring to would be Intention for the ESL or non-blessed within the ESL but still Intentionally for end-users.

[epage]

Have you talked with them for what lessons they have just doing a fraction of what this RFC does?

[oxalica]

Is there a possibility of blessing blessed.rs itself, making it into a semi-or-fully-official list? I think it's another alternative solution worth mentioning.

This would avoid the cost of maintaining crates themselves, bless multiple crates for different design goals. Newborn crates competing existing crates (with new design, new goals, new algorithm or etc) can also be less painful (as mentioned by @jswrenn https://github.com/rust-lang/rfcs/pull/3810/files#r2080592616) and they can be added to (not replace!) the list later if some conditions are met.

[epage]

As noted in https://github.com/rust-lang/rfcs/pull/3810/files#r2080548607 this runs counter to that RFC as that is about a cohesive API while this is about an organization.

[epage]

What lessons can we learn from these prior art?

[erickt]

Should this RFC discuss if it will be possible to remove crates from the ESL, or would we allow for duplicate functionally? I recall stdx was hoping to support removing crates, but I imagine it could be very disruptive to the community.

[epage]

This would be longer support than we off Rust itself

(yes, this is an example)

[comex]

Not really. The text is talking about how long to support an old SemVer-major version after releasing a new one. Rust itself never has SemVer-major updates, so the issue never arises. Theoretically at least.

[epage]

While this is an example, it highlights the assumed maturity for ESL.

Is there a place for younger crates that fill a core need within ESL? I'm again thinking of my libtest work once its further along.

[ssokolow]

...and what role does rust-lang-nursery play in this vision? I'm not sure it's even easy for people to learn what membership in it means now.

[epage]

Not to debate policy but just so I don't forget, there might want to be policies on this. In most cases, there is little difference between minor and update. When supporting for 2 years, there are major differences

• The more majors, the more work to backport, patch, and release
• Without majors, you can't patch. MSRV bumps and "risky" changes (major refactors) should necesisitate a minor

[Lokathor]

I think we should not accept pre-1.0 crates into the ESL. I'm not even joking.

[riking]

Being on a 0.x.x version hampers so much of your ability to perform semver updates; imposing a technical requirement for a non-0.x version does seem justified.

[epage]

Not to debate policy but just so I don't forget, consider the MSRV applying to the package and not every version.

What I mean is that a previous but supported minor satisfying the MSRV should count, even if the latest version doesn't. Maintainers will have flexibility to bump MSRV while having an incentive to not do it too often because it means more versions to support, creating more work, see https://github.com/rust-lang/rfcs/pull/3810/files#r2080681065

[epage]

Similarly about not wanting to forget, consider whether "N versions backs" actually meets people's needs. That means the MSRV can change every 6 weeks. Those with a direct MSRV requirement usually need predictable, stable stuff. Calendar based approaches might work better.

[Lokathor]

If were legislating MSRV we'd need to actually pick a policy on cargo features affecting MSRV, which means the cargo resolver should support that logic as well. Not that we can't do any of that, but it's complicated

[epage]

From where I am currently out, I see per-feature MSRVs to be highly unlikely.

[Lokathor]

If that can't be supported in some way, then "two years of msrv" is effectively a joke proposal.

[Lokathor]

epage was just taking notes on what the ESL org would have to cover eventually, I'm mostly just adding to the notes pile.

and i know that RFCs aren't always implemented exactly as first written, but if there's not really a remotely realistic way to implement what the RFC says, then the RFC needs to say something else. The current RFC text currently states a pretty specific policy.

[BurntSushi]

Okay.

[epage]

Along the lines of MSRVs is the conflict between long MSRVs and build times which is also another strong care about. I was analyzing someones dependencies recently and there were a lot of deps that exist solely for MSRV. Some could be handled through polyfills while others will require cfg(version) / cfg(accessible) to be stabilized. This is another reason to consider MSRV to be about the project as a whole and not apply to every major or minor version.

[nicoburns]

no_std support is another major source of otherwise unnecessary dependencies. Many projects unconditionally depend on libraries like libm and hashbrown because there is no ergonomic way to specify dependencies that are only enabled when not(feature = "std"). So you either have to add something like a no_std or libm feature (which is inconvenient for consumers of your crate) or unconditionally depend on the necessary crates.

[epage]

I wonder what all cases there are like this and if there are some high leverage changes that could be made to avoid this, like getting HashMap into alloc with a no-std hasher available.

[epage]

Not to debate policy but just so I don't forget, we should consider a little more nuance here.

Problems with releasing new major versions for why they should be throttled / avoided

• Work to upgrade
• Interop when in a public API of another package
• Compile times from multiple copies

Not everything is big and not everything is meant to be in a public API. As for being disruptive, we should probably encourage (but not require) breaking releases to be removal of deprecated functionality as discussed on matklad's blog and as I try to do with clap and winnow as much as possible.

[Lokathor]

I don't think I understand what you're trying to get at here. When you say "major" do you mean the first number in the three number sequence? cargo's vision of versions is to treat the first non-zero version as the "breaking change" number, and the next numbers as compatible, right? So how is 0.x.y somehow better than x.y.z? An incompatible version is just incompatible.

[epage]

The topic I'm replying to and my comment and "major" to refer to the relative position in the version number and not the absolute position (wish we had established names for these)

My comment is speaking to how to manage breaking releases.

[epage]

I think I saw where some of the confusion came from and made an edit that I hope helps.

[epage]

What about non-malicious but unfavored changes?

This can range from

• A maintainer making the API more non-ergonic for purity purposes (highly debateable on whether this is a problem)
• A maintainer shipping pre-built binaries (we saw how that went)

Flipping the perspective on this, I know with clap I've appreciated having WG-CLI as a sounding board for things I thought might be a good idea but I was unsure how they would be received.

[ssokolow]

*nod* APIs becoming less ergonomic is definitely a concern.

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths (the only reason I choose it over gumdrop's offering of a derive API with faster build times as I do in cases where I can just say "non-UTF-8 paths not supported" like actix-web www roots and config file paths) so opaque last time I looked at it that I've built a habit of just rg-ing my way to one of my existing projects which contains a suitable OsString-processing pipeline and copy-pasting the relevant snippet to the new project without even trying to remember why it works anymore.

Clap 3's "vendored StructOpt" API? Comprehensible. Clap 4's overindulgence in rustdoc-hostile, rustc-error-message-hostile type-level dispatch obfuscated by weaknesses in the concept of derive macros? Not really.

For me, clap's API has become a bag of incantations with my prime use-case for it achieved through a piece of tribal knowledge that you blessed me with in some comment on the issue tracker that I forgot to bookmark... and I'm one of those people who makes an active effort to understand APIs before I use them and to maintain that understanding. (eg. I "make things harder than necessary" when implementing custom Serde work by trying to save the brute-force "just impl a whole new Serialize" option as a last resort until I've proven to myself that there's no higher-level way you're meant to be able to do something.)

[epage]

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths

While a bit off topic for this RFC, OsString paths "just work" which was a large reason for the clap v4 design as before where you had to know they weren't supported natively and find the right incantation to get them to work.

[ssokolow]

They "just work(ed)" if you "just want an OsString". Last time I wasn't just copy-pasting a previous successful use, they were a frustrating and under-documented type-system puzzle-box if you wanted to plumb in some custom validation and end up with a PathBuf field on your struct. (But, as soon as you told me the magic incantation, they did indeed "just work".)

With StructOpt and Clap v3, I had no trouble understanding from the docs how to specify that I wanted the argument ingested as an OsString, passed to a validator function, and then ending up in a PathBuf... even if the APIs were suboptimal.

[oxalica]

What about non-malicious but unfavored changes?

There is already an example in the ecosystem,though not quite in this RFC's scope, gluon-lang/lsp-types#284. Some downstream users are so unfavor the changes and start creating forks. I think it's a lose-lose for the ecosystem. The problem will be worse if the crate in question is in ESL.

[Kijewski]

Maybe you could indent the content under a bullet point to make it more readble?

1. **Title**

   Description

   More description

2. **Next point**

   It has a description, too.

1. Title

   Description

   More description

2. Next point

   It has a description, too.

[nicoburns]

I'm glad to see someone trying to address the problem of supply chain attacks. I think it's an important issue which is currently very difficult to mitigate against.

However I do not think that the presented solution of centralising maintenance of key crates is a good one. I agree with @clarfonthey that the constraint which creates the opportunity for supply chain attacks is lack of available maintainer resources, and that this proposal is likely to reduce the availability of such resources not increase them.

I think the philosophy behind Cargo Vet where vetting/auditing is decoupled from maintaining/owning is the right one, and if there is funding available, I would love to see this model expanded via things like:

• An official, well-maintained version of diff.rs. This website allows people to 1. easily read the code actually published to crates.io (not just the code checked into git which may be entirely different) and 2. see a diff between versions of that published code. To me, these are foundational capabilities for enabling audits.
• Vetting/auditing functionality built into this "diff.rs" UI (e.g. something similar to Github review UI) such that people can easily create audits/reviews via the UI. These would then be saved to a central database.
• Metadata attached to audits with things like standardised categories which denote the depth of review that has been undertaken and the reviewer's confidence level in their review.
• Tooling built-in to cargo which allows users to specify criteria such an "N reviewers (with an optional allowlist of specific reviewers or organisations) must have approved" and/or "there must be less than N negative audits" (perhaps with an additional "warn" outcome that prompts the user to go and read the audits and/or audit the code themselves).
• Pushing crate authors towards MFA
• Expanded support for (emphasis on) signed crate releases (e.g. support for allowlisting certain crates only if new releases are signed by a trusted signing key and prompting for manual verification if a new key is used)
• Directly funding trusted people to spend their time producing audits

Trusting specific crates/publishers is all well and good, but the supply chain attacks I would be most concerned about are when crate maintenance changes hands or a users credentials are compromised. To mitigate against these kind of attacks one really needs each published version to be verified, not just the crate or it's publishers. Given how many users of the really key crates there are, it should be easy to get enough reviews if making one is easy enough.

I think lists of approved crates/authors/auditors/organisations would be useful. But I see no reason why there needs to be a single list. We could perhaps have an "official" rust foundation list that is quite conservative. But also allow other organisations to publish lists which people might decide to either trust wholesale and pick and choose pieces of to form their own lists (e.g. corporations like Mozilla, Google, Amazon, Dropbox, etc; open source collectives like Servo, Linebender, Bevy, etc; and even individuals that may be trusted by the community).

[epage]

I think I realized what was missing in this RFC that has led to some of my scattered comments: a lack of guiding principles that the ESL team is working off of that helps setup the expectations / contract between

• this team and the target audience (one off developers? companies?)
• the project
• the community

[epage]

A comparison with Boost might be relevant as it kind of acts like this

[kennytm]

I don't see how any of these 5 are "similar effort" to the proposed RFC. These are just "battery-included" library which evolve together with the language. ESLs are updated independently (and more importantly for Rust, allows breaking changes "v2.0"). I think these are more relevant:

• C++ Boost
• Haskell platform (which has been deprecated)
• golang.org/x/*

[epage]

I think something missing is an expected outcome driving this.

Do we expect N% of X type of applications to be developed with only std and the ESL? How much past the ESL will they go? Does that negate the benefit since they still have to deal with a large web of trust and figure out which dependency to use?

[epage]

I am very sympathetic to the problem that this is trying to solve.

We recognize the failure of a batteries included std so we rely on the "market" to fix it. This has its benefits but there are also market failures, particularly the burden it puts on "consumers". The cornucopia of options can be overwhelming and put someone in analysis paralysis. This is made worst by the fact that most people needing to decide what to use are likely not qualified to decide what to use and they need guidance. We can provide supply chain management tools like cargo vet but they still require each person to "do their own research".

Providing users with "good enough" starting points that they can diverge from when they encounter specific constraints would be a big help to developers. Standardizing best practices among those (remember the lib blitz?) can help ensure some basic quality and consistency.

I think experimenting in this general direction while also working to improve the overall supply-chain story are good. I worry though that this is trying to be too ambitious in its goals and that (1) this will put too large of a burden on maintainers that participate and (2) this will look like its solving more of the problem than it will and it won't even be able to solve as much of the problem as this comes across.

Overall, I'd prefer to see a softer take, treating this as an experiment, evaluating what the most important goals are and focusing on those, and developing guiding principles for the operation of this experiment. By being an experiment, it does limit the trust element and requires it to be more easily rolled back but I think thats good.

In parallel to this, I would like to see more investment on supply chain story. I know the Founation has talked about efforts several times but I feel no progress is being made. Maybe thats due to how communication is working or because their priorities are different than mine (which can be ok).

[epage]

While I'm thinking of it, something that should be considered for the policy is when packages transition out of the ESL (at the ESL's choice at least), that I assume to meet the intent of this policy, that major versions released while in the ESL would continue to operate under ESL policy, including 2 years of support.

[pbdeuchler]

While this effort aligns with my own personal opinions about language ecosystems I'm unsure if this is a realistic way to achieve the stated goals.

One of the biggest difficulties when dealing with supply chain security is the asymmetric nature of it. It only takes a single, well crafted, compromised file of code to breach an organization. The vast majority of risk comes from the long tail of 3rd party dependencies and their transitive dependencies. As a security engineer I am far more concerned in my day to day about random helper libraries that touch I/O (say, log rotation) than I am about serde or tokio. I understand that compromise of those large, central dependencies is a worst case scenario with a massive blast radius but I am unconvinced that simply hoisting that functionality under a blessed namespace provides any hard security benefits over the existing status quo. Security is often a function of raw resources dedicated to a problem and the existing eyeballs, momentum, and man power most large rust libraries wield will be a significant hurdle to overcome, at least initially.

That said, I think this RFC does itself a disfavor conflating the efforts to bring, say, encoding support into a blessed implementation with things like ser/de and HTTP and especially so mixing them within the same tranche. Not only will these have differing implementation timelines moving forward but I think the security benefits for blessing some of the "lower level" primitives massively outweigh the benefits of others. This is mostly for a single reason: the transitive dependency explosion that is somewhat unique to Rust. The dependency graphs for most languages (or at least most languages I have experience with), usually start to circle back into the standard library (or at least a defacto standard library) once you go a couple layers deep. This is not true with Rust and is patently obvious whenever you build something non-trivial as even a straightforward project can easily explode to hundreds of total dependencies.

I would advocate this RFC is dramatically scoped down to the following:

* ‘syn’ equivalent
* ‘bitflags’ equivalent
* Encoding support (base64, hex, etc)
* Random number generation
* Error handling support

### Tranche 2: Next step up in complexity/abstraction
* Regular expressions
* Compression (e.g. zlib, zstd) <--- Very much on the fence with this one

In my opinion this is still a very ambitious list and could still be scoped down much further.

I believe these provide the largest cost/benefit ratio for improving supply chain security within the rust ecosystem and success with this list would increase confidence that larger projects could be taken on. I also believe that picking winners in these specific domains will be much less contentious and more straightforward. Last, most of the efforts within this scoped down list have a known definition of done. The scope around providing a default async runtime or HTTP libraries is massive. If my career has taught me anything it is that unwieldy and more importantly undefined scopes are the death of any ambitious software project. And while I think that something like a Rust equivalent to the Golang crypto standard library would be an incredible boon to the community at large I am skeptical that the rust foundation has the raw resources to complete just that single line item with the stated funding.

[pbdeuchler]

Separating this out into it's own comment because it might be much more controversial and I care far less about it:

I think absorbing these libraries into a blessed namespace but keeping them functionally out of the std library is a half measure that demonstrates a lack of conviction on the way forward. If we truly want to achieve many of the stated goals I think at least 75% of the list stated here (and the entirety of the list I provide above) should be within the std library proper. Anything else risks creating a scenario that's the worst of all worlds, where these "blessed" implementations bring all the baggage and bureaucracy of working with a language foundation that has many competing interests, with none of the benefits of working with a nimble quasi dictatorship that often exists within the FOSS world or being within the language itself. This isn't a value judgement, it is simply a fact that contributing to a std library (or an officially blessed quasi std library) will usually have a much higher barrier to entry for contribution than 99.999% of all FOSS projects. From a security point of view this is unavoidable, if we want to provide security guarantees around these libraries contributions that would have otherwise been accepted will have to be rejected. At that point you might as well make it formal.

[the8472]

I think absorbing these libraries into a blessed namespace but keeping them functionally out of the std library is a half measure that demonstrates a lack of conviction on the way forward. If we truly want to achieve many of the stated goals I think at least 75% of the list stated here (and the entirety of the list I provide above) should be within the std library proper

We don't want a std 2.0 (cf. python 2/3 breakage). Which means API stability requirements for inclusion in std are very strict and API removal can only happen in exceptional circumstances. If you look at cryptography then that's a constantly evolving field because some of it rests on foundations of sand. And while HTTP has high-level stable semantics its entire innards have been swapped twice in the last 10 years, and since rust is a low-level language we'd likely have to expose those to be useful in many contexts, this seems like too much churn for something that's supposed to be stable.

[scottmcm]

What about doing this needs an RFC? Couldn't interested parties just do this without having this RFC?

TBH, I suspect that one ESL is the wrong approach because it's going to end up getting overtaken by one specific subset of rust uses and thus poorly serve the other ones. The mostly likely outcome, to me, is that the well-funded people making web services will take this over, and we'll end up with "well obviously tokio and base64 are critical for everyone", even though we have tons of CLI things written in Rust that don't ever care about either.

I fully support people in a particular domain getting together to provide a holistic story for that domain with excellent security and ease of use, as well as other things like writing a book describing the paved path of using all these things together smoothly and in a way that avoids footguns. But I don't see why rust-the-project needs to bless those things, or why there should only be one of them.

[Manishearth]

This is also why I think doing this via cargo vet + potentially some select crates.io integration is a good idea.

Any group of people can band together, define standards, and certify a list of crates as following those standards. That's what the tool is about! As a community we can elevate certain sources of this type of thing to make it discoverable.

[ssokolow]

I think absorbing these libraries into a blessed namespace but keeping them functionally out of the std library is a half measure that demonstrates a lack of conviction on the way forward. If we truly want to achieve many of the stated goals I think at least 75% of the list stated here (and the entirety of the list I provide above) should be within the std library proper

We don't want a std 2.0 (cf. python 2/3 breakage). Which means API stability requirements for inclusion in std are very strict and API removal can only happen in exceptional circumstances. If you look at cryptography then that's a constantly evolving field because some of it rests on foundations of sand. And while HTTP has high-level stable semantics its entire innards have been swapped twice in the last 10 years, and since rust is a low-level language we'd likely have to expose those to be useful in many contexts, this seems like too much churn for something that's supposed to be stable.

Not to mention that in C++, stability requirements (in that case, ABI stability) led to std::regex being slower than shelling out to PHP.

We already have std::sync::mpsc in std. Do I ever use it? No, I used to use crossbeam-channel for the MPMC API that it still doesn't provide, despite now wrapping a vendored copy of that. Now? I use Flume because it also does async.

Whenever I see calls for a "batteries included" std, my intuition is that people are reversing cause and effect... thinking that putting things into std will make them stable, rather than that things having stabilized makes them a candidate for being in std.

It reminds me of how The Art of UNIX Programming from 2003 has parts that aged well (the things that were already solved problems) and parts that aged poorly (eg. Recommending XML for structured file formats because JSON and YAML weren't even two years old yet. Recommending a now-obscure protocol named BEEP for what XMPP seemed likely to do and now WebSockets are doing. Recommending DocBook XML for what is now being done by tools like Sphinx and mdBook, despite AsciiDoc providing a Markdown/ReST-like lightweight syntax that maps 1:1 to DocBook. etc.)

Making BEEP into RFC 3080 didn't drive adoption because standardization doesn't magically overrule all other concerns.

[pbdeuchler]

We don't want a std 2.0 (cf. python 2/3 breakage)

Not to mention that in C++, stability requirements (in that case, ABI stability) led to std::regex being slower than shelling out to PHP.
We already have std::sync::mpsc in std. Do I ever use it? No, I used to use crossbeam-channel for the MPMC API that it still doesn't provide, despite now wrapping a vendored copy of that. Now? I use Flume because it also does async.

There is always the option of not repeating the mistakes that languages have made in the past. I'm not trying to be trite but to me these aren't necessarily arguments against expanding or changing the std library, these are arguments against expanding or changing the std library without long term resource commitments and a well known plan. I would also caution people against conflating resource constraints with technical constraints. I do think whatever your thoughts on Go as a language, I think the Go team has conclusively proven that given enough resources and direction these are very surmountable challenges by just investing in quality maintenance work.

But that obviously doesn't actually address these concerns for us as the Rust Foundation has significantly fewer resources than Google. To me this is an even stronger argument for using inclusion to the std library as a litmus test. If the rust foundation doesn't want to commit resources to including something in the std library (or alternatively, they don't believe such a thing belongs there) then why are we committing resources to an ESL for that same code? If the answer is just security it's my personal opinion that sponsoring regular security audits would be infinitely more cost effective. Otherwise the ESL will incur large costs with very little benefit.

Please correct me you disagree here but again I just don't see a way these ESL libraries can achieve this middle ground where they are stable, trusted, reliable, and secure but still able to be nimble, make breaking changes, and iterate fast without dedicating roughly the amount of resources required to include that same code into the std library.

Edit: In case it's not clear, my general thrust here isn't that these things should be in the std library, just that long term blessing this code has roughly the same resource commitments as including them in the std library, and if we don't want to do that then we shouldn't include them in an ESL either.

[bdaehlie]

Regarding the question of whether this needs to be a Rust project thing, I don't think it does. The idea was to propose it to the Rust project first and if folks decide that doesn't make sense, then we can look at potentially organizing something outside of the Rust project.

I'm primarily concerned with mitigating the problem of there being too many dependency sources for the average Rust program. Whether or not this particular proposal is part of how that happens is not as important. If conversation about it leads to other improvements to the supply chain security problem that's a great outcome.

[arielb1]

I think this RFC is conflating 2 different concepts:

1. Is this library going to burn your house/company down?
2. Is this a good tool for usecase X?

In the real world, we have bodies like Underwriters Laboratories that focus on (1) while completely ignoring (2).

These are very different concepts. A library that has a hard-to-use API is not going to burn your house down. If there are 2 competing libraries, one of which is popular for projects started 5 years ago, and another one is popular for projects started 5 weeks ago, it makes sense to certify that both libraries will not start fires, even tho one of them is probably not recommended for new projects.

[As a separate problem, if a human group with an horse in the race, such as the Rust Project, is going to start putting stamps on libraries, this will cause drama for well-known human reasons].

@pbdeuchler

For std inclusion - the Rust std is closer to Python's builtins than to Python's standard libraries. std is versioned along with the compiler, which makes it annoying for people to control updates of it, which means that for everything that does not want to "live with the compiler", it probably makes more sense for it to live in a rust-lang/X non-std library than in std

[nicoburns]

Please correct me you disagree here but again I just don't see a way these ESL libraries can achieve this middle ground where they are stable, trusted, reliable, and secure but still able to be nimble, make breaking changes, and iterate fast without dedicating roughly the amount of resources required to include that same code into the std library.

@pbdeuchler I think the idea is to have a set of libraries that are "trusted, reliable, and secure" but that are not 100% stable and can make breaking changes. Probably these libraries wouldn't be really iterate fast or nimbly but they would be able to make breaking changes with major version release if needed.

[the8472]

I'm primarily concerned with mitigating the problem of there being too many dependency sources for the average Rust program.

This seems like rearranging the deckchairs on the titanic musical chairs. There are finite resources to review things. A lot of focus already is on the core ecosystem. Formal verification, fuzzing and all that. And it's still not enough. And yet every big project will need stuff outside the core set. Maybe an OIDC client or some SNMP or XMLRPC parser (ugh). In the end everything ought to be reviewed. The number of people it comes from doesn't really change much about that.

[ctz]

I think absorbing these libraries into a blessed namespace but keeping them functionally out of the std library is a half measure that demonstrates a lack of conviction on the way forward.

Just picking up on this point, I think it's important to recognise the current architectural reality of std:

• it cannot be built by its users,
• is not a crate so cannot be [patch]ed, shimmed, poly-filled, ported, or forked,
• is not separately versioned and so can never break,
• has a weird and unique relationship with its compiler,
• has a weird and unique relationship with its dependencies,
• is supported on stable platforms that it arguably shouldn't be (eg, functions that just panic statically, are not documented to do that, and can never do anything different),

I realise some of those issues are being worked on now, and I hope that work is successful.

But the idea of permanently stabilising things into std up to the complexity of a HTTPS client seems like a non-starter on this backdrop. And I think that's why this RFC does not even go near that idea.

[BurntSushi]

But the idea of permanently stabilising things into std up to the complexity of a HTTPS client seems like a non-starter on this backdrop. And I think that's why this RFC does not even go near that idea.

As a libs-api member, I very strongly agree. I don't ever see "HTTPS client" as being a viable thing that will make it into the Rust standard library.

To a first approximation, APIs added to std have to live forever. It is very easy to under-estimate the profound impact that has on API design and what we're willing to add to it.

[zdivelbiss]

If you want a standard library with every feature you can imagine, then you should use Python, C#, Java, or any number of more high-level languages. A systems-level language like Rust has no business trying to package things so far outside core functionality in its standard library. There are already places in the standard library where certain trade-offs are made that favor one use-case over another—to the detriment of overall development in that use-case.

Just my 2¢.

[pbdeuchler]

@arielb1 @nicoburns I think we are all in violent agreement here... I'm not advocating for putting unstable APIs in the std library. But 75% of the domains in this RFC are well known and stable, those I would advocate for putting in the std library. The remaining 25% that are external "blessed" libraries, however, will still require effort commensurate with any std library work. The rust foundation should be ready to commit to spending those resources long term if it wants to go down this path (which I support).

But regardless I am worried I have created a thread of conversation that distracts from the main proposal, I did not wish to re-ignite a debate about std library inclusion... happy to concede this point in that case.

[ChrisDenton]

I feel this needs a deeper exploration of the alternatives. I think most previous efforts have suffered from a lack of funding, visibility and/or trust. But I think things can be learned from the current ecosystem and past failures. The support of the foundation and others would greatly help.

lib.rs shows that a curated front end is viable. Anyone can set up such a front end and limit it to just blessed crates. The "rust-bus" initiative, where crates would add that organisation as a co-maintainer. was a good idea. It didn't work out at the time but with appropriate funding and buy-in then maybe it could, especially as part of a larger effort.

Putting it together, a blessed front-end and an organisation that can co-maintain crates would, I think, go a long way to addressing the motivation for this RFC without needing more drastic changes. Trust, crate discovery and maintainer support could all be addressed (or at least substantially improved) without being so invasive. The organisation can set whatever terms it likes for crates to be included.

[fintelia]

This might be over-fitting to the xz attack, and in any case is likely to be exceedingly controversial...

I count at least two libraries for zstd and six(!) for zlib [1, 2, 3, 4, 5, 6] that all currently get at least half a million monthly downloads. Plus another crate that provides a common abstraction layer over 3 of the 6 zlib crates

[joshka]

One thing that struck me when seeing the ESL acronym is that it ESL often stands for "English as a Second Language" in a large number of English speaking countries. This may not be the sort of acronym overload that you want for a project like this. Either Stdx or StdEx (more aligned with the naming of rust extension traits) both seem like like much better names for this.

I've intentionally added this comment here near the end of the document rather than at the top of the page where ESL is first introduced. Hopefully this will avoid too much of a bikeshed discussion happening and distracting from the much more substantive conversations.

[ssokolow]

We don't want a std 2.0 (cf. python 2/3 breakage)

Not to mention that in C++, stability requirements (in that case, ABI stability) led to std::regex being slower than shelling out to PHP.
We already have std::sync::mpsc in std. Do I ever use it? No, I used to use crossbeam-channel for the MPMC API that it still doesn't provide, despite now wrapping a vendored copy of that. Now? I use Flume because it also does async.

There is always the option of not repeating the mistakes that languages have made in the past. I'm not trying to be trite but to me these aren't necessarily arguments against expanding or changing the std library,

The problem is that clairvoyance is an unsolved problem. Being extremely conservative about how much stuff is baked into the library you've committed to never having a "Python2 to Python3" transition on is "not repeating the mistakes that languages have made in the past".

these are arguments against expanding or changing the std library without long term resource commitments and a well known plan. I would also caution people against conflating resource constraints with technical constraints. I do think whatever your thoughts on Go as a language, I think the Go team has conclusively proven that given enough resources and direction these are very surmountable challenges by just investing in quality maintenance work.

Personally, I think at least half of Go's solution to it being an insurmountable problem is WONTFIX NOTABUG-ing the problem out of existence. It's much easier to design an API which lasts if you just define the unsolved parts as out-of-scope.

Go is what you get when people think all C needed is garbage-collection, a perforated implementation of memory safety, a concurrency model that was all the rage in the 1990s, and a strong anti-FFI belief that "cgo is not Go" because things like extending Go-style easy cross-compilation to FFI is hard work that requires cooperation with people like the LLVM or GCC developers.

But that obviously doesn't actually address these concerns for us as the Rust Foundation has significantly fewer resources than Google. To me this is an even stronger argument for using inclusion to the std library as a litmus test. If the rust foundation doesn't want to commit resources to including something in the std library (or alternatively, they don't believe such a thing belongs there) then why are we committing resources to an ESL for that same code? If the answer is just security it's my personal opinion that sponsoring regular security audits would be infinitely more cost effective. Otherwise the ESL will incur large costs with very little benefit.

Please correct me you disagree here but again I just don't see a way these ESL libraries can achieve this middle ground where they are stable, trusted, reliable, and secure but still able to be nimble, make breaking changes, and iterate fast without dedicating roughly the amount of resources required to include that same code into the std library.

I can agree with that. Every time some kind of stdx has come up, my perspective, has always been "OK... but where is this extra manpower gonna come from?"

Edit: In case it's not clear, my general thrust here isn't that these things should be in the std library, just that long term blessing this code has roughly the same resource commitments as including them in the std library, and if we don't want to do that then we shouldn't include them in an ESL either.

I disagree that they should be in the standard library for the same reason that, when you look at things like accessing members of collections, C++ reveals itself to be three different languages in a trench coat, refusing to admit that C++ failed and C++ v2.0 failed by disguising the fact that they underwent a Python2 vs. Python3 vs. Python4 transition as, instead, a "Python 2.x has urllib and urllib2 in the standard library" situation.

[Diggsey]

I think there's a nugget of value in this RFC: specifically a set of "higher standards" that crate authors can choose to follow, where those standards are agreed upon in a centralized way.

These parts on the other hand seem actively harmful:

• Moving crates to a central github org is pointless and creates a single point of failure.
• Naming crates after ESL creates harmful churn for users of the crates, and the idea that crates will be created to be ESL crates from the start is pure hubris, and will lead to a ton of garbage ESL crates.

I think what could work would be a decentralised trust model:

• Any organisation can announce on a crate's crates.io page "I trust this crate revision". The amount of scrutiny in this review would be entirely up to the organisation.
• As a user, I can use a cargo audit like tool to find dependencies that are not trusted by any of a set of reviewers that I trust.

With this approach, any person or organisation can create a "brand" around their crate reviews.

Optionally, a Rust ESL team could maintain a list of "trusted reviewers". They would set the criteria for what warrants inclusion in this list, but might include:

• Having a certain number of reviews.
• Being independent of other reviewers in the list.
• Having certain standards of review.
• etc.

Following from that, any crate revision which has been given the seal of approval by at least two or three different reviewers on this list would be eligable to be part of the ESL.

Notably in this setup, the ESL team is concerned only with policy making and maintaining this reviewers list, and is freed from the more time consuming work of reviewing the many crates involved. Furthermore, it gives large companies which might already be doing some level of review of their dependencies the ability to feed back that value into the ecosystem, without ESL crates being exclusively corporate focused.

[mcobzarenco]

I see a philosophical problem with blessing a subset of dependencies for writing backend code. Rust will suddenly look a lot less like an all-encompassing replacement for C/C++ and instead more like a supped up Go for writing APIs. I don't want to see this.

Just to pick a random one, why have base64 in std? I have been writing Rust since 2013 (since 2016 professionally), and I haven't serialized/deserialized something as base64 once for work.

To my machine learning/scientific computing eye, that's something a backend server cares about. Not to mention, there are multiple base64 libraries with different trade-offs.

[arielb1]

I see a philosophical problem with blessing a subset of dependencies for writing backend code. Rust will suddenly look a lot less like an all-encompassing replacement for C/C++ and instead more like a supped up Go for writing APIs. I don't want to see this.

I think that much of it is that backend servers are where the "Underwriters Laboratories" stamp is most important. For other sorts of programs there are generally other methods that prevent them from burning down your company (people might also want a "safe for embedded" stamp, but that stamp would have different regulatory characteristics than a "safe for backend" stamp).

But I agree that the existence of multiple versions of base64 decoders (my fairly simple Cargo.lock has 3, 4 if you count different versions of base64) is a pretty good argument against having a decoder in libstd, but also a pretty good argument for having a way to "bless" them all.

[VorpalBlade]

I posted this originally over on the reddit thread but I think I should cross-post it here so it comes to the attention of those working on this RFC:

I have a number of concerns with this RFC:

1. It will entrench a number of "local optima" crates even further. For example, serde has flaws: it cannot support true zero copy deserialisation (e.g. mmap file and be able to use it directly as is) without breaking changes that are unlikely to happen. And due to monomorphisation it can bloat the code size and build times (problem for embedded and for projects where build time matter more). The crate Facet by fasterthanlime seems like a really interesting alternative approach that could solve this, but it is in fairly early development still. And don't get me started on tokio, io-uring and non-server async. The non-duplicative clause is especially bad here as it stifles innovation.
2. Just because it is officially blessed won't mean it actually gets maintained. Python had issues with this, where they had to remove standard library modules that went unmaintained (smtp support and some now obscure file format if I remember correctly for example). And in some cases better alternatives came out in the ecosystem and people stopped using the standard library modules (and for those not in the know, they are now using worse modules). In python you shpuld use regex instead of re, requests instead of http.client, pytest instead of unittest. There is a saying in the Python community that "the standard library is where libraries go to die". I fear this would happen here to.
3. Syn which is mentioned in the RFC is slow to compile, surely we can do better, such as unsynn? This is related to the point here.
4. Which hex or base64 crate? The one that compiles to small code (yay, good for embedded and wasm) or the one with SIMD support (good for my 128 core server load where only speed matters).
5. Also, Rustc itself and std depends on multiple crates from the ecosystem. And while they do check those, it is not a like they do a full audit of every dependency, and especially not on every update. How would that interact with this ESL idea? Would rustc only be allowed to use dependencies from ESL?

[arielb1]

I agree. I expect the Rust project to have negative alpha when it comes to finding "the best crate for usecase X". Also, I believe that "find the best crate for usecase X" is best done by an algorithm - such as lib.rs, or a search engine/LLM. It might make sense for rust-lang.org to link to one.

Which does not detract from the Rust community benefiting from a way of knowing that a crate is "reliable", even if not necessarily "the right tool for a particular job", along with a path for crate authors to be able to get their crates to reliable (of course, the Rust Project considering a crate to be "reliable" does not mean that it is owned by the Rust Project. For example, I am quite sure that the Rust Project regards Tokio as a highly reliable crate, but Tokio is run by a separate project).

[abgros]

Will the extended standard library be automatically downloaded by Rustup?

[arielb1]

Moving crates to a central github org is pointless and creates a single point of failure.

Giving maintainers of popular crates the option to put their code in a Github repository with certifiably high security standards and coupling the code on git to the code on crates.io somehow (while letting the maintainer have complete control over the code except in "actual security breach" situations. I mean, Github owns the git repository and the Rust Foundation owns the crates.io index even today and they might act even today in the case of an actual security breach) rather than having maintainers needing to figure out by themselves how to secure github.com/jsmith123/my-crate, sounds to me like a good enough idea if it can be done in a reasonable way.

Of course, if maintainers already have some sort of git repository that they know how to secure, they might not need this help, and the Rust Project should not have a "let us own your repo OR ELSE you won't get a pretty badge" system.