Add support for --workspaces to run on all npm workspaces

[voxpelli]

I'm thinking it would be neat to be able to do:

run-p --workspaces check:*

Related issues:

• Support a --workspaces and maybe --workspace flag to run in a npm workspace monorepo voxpelli/node-installed-check#31
• Support a --workspaces and maybe --workspace flag to run in a npm workspace monorepo dependency-check-team/dependency-check#188

Interesting modules:

• https://github.com/voxpelli/read-workspaces
• https://www.npmjs.com/package/@npmcli/map-workspaces
• https://www.npmjs.com/package/yarn-changed-workspaces

[bcomnes]

Makes sense now that npm supports workspaces.

[voxpelli]

When --max-parallel is set, should it try to spread out the executions across the workspaces or just assemble a long list of all tasks to be run and execute them from top to bottom?

The latter would be more a 1:1 between run-s --workspaces and run-p --workspaces but I kind of like the idea of balancing it across the different workspaces. Probably overengineering to do though, right?

[bcomnes]

Probably do what ever is simplest to implement and maintain.

[voxpelli]

I'm going to give it a stab now

[bcomnes]

If you do take a stab at this, feel free to refactor ancient promise callbacks to async await etc. Whatever cleans it up.

[voxpelli]

Created a helper for reading all packages from a workspace, with similar filtering as npm itself has: https://github.com/voxpelli/read-workspaces

Started using it in list-installed and as such also in installed-check / installed-check-core now

[voxpelli]

RFC for providing npm-run-all style functionality within npm itself, with workspace support being one of the highlights: npm/rfcs#190

Maybe we could try crowdfunding the development of this addition? Through eg. https://polar.sh/ or Open Collective?

[bcomnes]

Would love for run-s and run-p to be upstreamed. Is npm accepting PRs these days?

[justinfagnani]

I've been advocating for incorporating Wireit's script runner into npm - since I don't think that running scripts sequentially or in parallel is nearly sufficient.

I did hear from an npm team member after filing that issue that while they loved the idea, they didn't have the bandwidth to maintain such an addition, even if it were community contributed. I'm not sure if that situation still stands. A development since then is that Deno is adding a Wireit-inspired script runner as Deno Tasks. Maybe that moves forward the recognition that we need a more robust script running solution.

[bcomnes]

I haven't used wireit yet, but it looks cool, especially for more sophisticated builds. Will give it a read through soon. run-s and run-p definitely have their limits, but also so does nearly every other tool in the npm ecosystem. Small modules almost never needed anything beyond what these two offered, but I also realize the trend these days is to try and operate entire companies out of workspaces, and so these simpler solutions fall over at that scale.

Personally I like working within limits, I think they help avoid large problems/messes but I understand this is completely the minority position these days.

[voxpelli]

With more elaborate approaches the design choices also increase and it’s harder to align everyone on the same approach

It’s the classic small core vs big core argument – should it be solved within the core or does it belong in a pluralistic ecosystem

I think run-p and run-s style functionality could maybe belong in core but more than that: Probably not

[justinfagnani]

I don't think you need much scale before run-p and run-s fall over - you only need 4 scripts to create a diamond dependency where the deepest script should only run once.

A common setup might be a test script that depends on test:unit and test:integration, which both depend on build.

[voxpelli]

run-s build test:*

[justinfagnani]

@voxpelli the goal is to be able to not have to know how the script graph is structured in order to correctly run the scripts.

If something like this were included in npm, I would expect that npm test would just do the right thing and that the config for test wouldn't have to list out its transitive dependencies.

[voxpelli]

@justinfagnani If you only have the 4 scripts then it works perfectly fine to do:

"test": "run-s build test:*"

There is some point where it may be better for tasks to describe their own dependencies, but what percentage of npm projects are at such a complexity?

Many even find it enough to do so something like eslint && mocha or similar and use a single script for everything