Use dependency of dependency when package is not in package set

[flip111]

When i want to use this library as dependency of my project https://github.com/srghma/purescript-tidy-codegen/blob/main/spago.yaml

i get

» spago run --backend-args '--help'
Reading Spago workspace configuration...

✓ Selecting package to build: html2halogen

Cloning https://github.com/srghma/purescript-tidy-codegen.git

✘ The following packages do not exist in your package set:
  - dodo-printer
  - tidy (did you mean: tldr, midi)

Since these packages are not in the package set can spago not just use the specified git repos, this one for example? https://github.com/srghma/purescript-tidy-codegen/blob/main/spago.yaml#L62 seems a bit redundant that i have to copy this manually into my spago.yaml file

[f-f]

That would entail reading the workspace section of the dependencies - there are a few issues with that and we're likely not going to implement it:

• the format for that section might change and we don't guarantee compatibility - we only read the package.dependencies key today, and that is guaranteed to stay stable because the Registry depends on it
• the package set for your current package becomes less declarative and more indirect, if we had this you would have to climb an unspecified amount of repos to reconstruct your package set, which goes against the self-contained nature of package sets
• ultimately the version that you might fetch from there might be fine for the upstream library that you're using but might not be the right one for your project. We can't know that beforehand so we avoid any automation in this regard

[flip111]

I try to understand what you wrote.

• the format for that section might change and we don't guarantee compatibility

Does it mean parts of the configuration format is stable and others are not? Might this change in the future with for example configuration scheme versions with possible backwards compatibility?

• you would have to climb an unspecified amount of repos

Wouldn't it be specified in the configuration files of the dependencies? I mean you would know how many lookups there are when you start looking it up, if that sounds problematic i guess a hardlimit with spago flag as override could be used?

• which goes against the self-contained nature of package sets

I don't understand the nature of package sets. All i know is from my previous experience with other programming languages (npm/js and haskell to know a few) which will just fetch dependencies of dependencies. Maybe you can explain some benefits of package sets? I'm not understanding what we are winning here

• but might not be the right one for your project

In case you decide to use the same package for your project as a direct dependency. And on top of that use a different version. Wouldn't it be obvious that there is a version conflict due to it being a different branch or different commit? I mean git is supposed to tell you what version you are on so it should be able to see if two copies of the same repository are on the same version or not. If there is a version conflict at that point there shouldn't be any further automation IMO. You can just leave it to the user to go fix that. I'm familiar with this mechanism with, for example, cabal. But i know that other programming languages also have a package manager that can point to conflicting dependencies. What further did you have in mind to automate here?

[f-f]

Does it mean parts of the configuration format is stable and others are not? Might this change in the future with for example configuration scheme versions with possible backwards compatibility?

It's more like a guarantee that some key parts of the config won't ever change. We won't guarantee the whole schema, otherwise we wouldn't be able to ever change anything.
If we do change something we'll try to keep backwards compatibility, and have schema migrations.

Wouldn't it be specified in the configuration files of the dependencies? I mean you would know how many lookups there are when you start looking it up, if that sounds problematic i guess a hardlimit with spago flag as override could be used?

Lookups are not the issue. The issue is about not having all the information in a single place that is easily accessible and debuggable. If we'd do this automatically and your package wouldn't build as a result of one of these transitive dependencies being broken, we would have to report the whole chain of dependencies, etc. Feels like it would be a worse UX.

Maybe you can explain some benefits of package sets? I'm not understanding what we are winning here

I have written about it in the docs. To summarise, package sets are similar to lockfiles, but more generic in the sense that a project might (and most often will) choose to only use a subset of the package versions listed there.

[flip111]

• which goes against the self-contained nature of package sets

As i read from the readme The point of a package set is to provide a "stable" set of packages that you can use to build your project, without worrying about version conflicts.. This already starts to break as soon as you start to declare your own extraPackages with their specific git branch. I don't understand your argument of when you lookup dependencies of dependencies that it would go against the nature of package sets since this guarantee is already out the window.

Feels like it would be a worse UX.

Maybe we should measure the time a developer needs to take with and without this feature. Imagine a big project with lots of dependencies and deep dependencies so that the effect is exaggerated.

With the feature you would declare your dependencies in extraPackages (as is now) and it would also resolve their dependencies and in case of conflict you would get a list of conflicts with the exact location of the problem by running a single spago command. You can then immediately start to resolve the conflict which is the same in both scenarios (second scenario down below).

Without the feature for each package. You already have your direct (first degree) dependencies in your spago.yaml. Now you have to:

1. go to google to find the purescript package on github
2. go to github and find the spago.yaml file
3. open your spago.yaml file in the editor and make a new section for the package. Copying git link and ref etc
4. copy the package dependencies into your spago.yaml

Repeat step 1 to 4 for all second degree dependencies.

5. run spago again to see if you missed anything or if it's good

Repeat step 1 to 4 for all third degree dependencies.

etc. until you manually resolved everything yourself

This is just for adding a package. When you want to remove a package it's even worse when you have diamond shaped dependencies. Like A depends on B, A depends on C, B depends on D, C depends on D. You want to remove B, so you want to remove D as well. Now you have two choices either you check all your dependency tree manually up front. Or second choice you just remove D and put it back if things start to break again.

Another thing that i don't like is that the dependency tree is no longer visible from the flat listing of extraPackages in your project.yaml. Maybe yaml comments can help but it's also a PITA to maintain the comments. I'm generally a big proponent for keeping everything in one place, though in the case the hierarchy is not viewable due to the flat file format, and this makes it worse.

The bigger the project the more labor intense and more error prone this becomes. I guess a missing dependency would get caught, so it's mostly about unneeded dependencies or your understanding what you have to change where since you lost track of the chain.

I find the UX argument not convincing sorry. Also i don't know any other programming language package manager that does it this way. Most of them resolve the dependency tree automatically. Either there are other languages that do it this way that i don't know of or all of them had a good reason to automatically resolve the dependency tree.

If you live within the package set of the registry, life is great. But there are a bunch of reasons you don't have your packages in there, and those use cases also need to be supported.

I don't mean to criticize because i think it would be great for spago users to have this feature. Let's hope in the coming years purescript gets used more and project get bigger so that people would benefit of having something like this.

[f-f]

I feel like the discussion is focusing on using spago as a tool to pull in git repositories - the scope that we cover is much bigger than that, and honestly I don't think I should personally spend time polishing the UX for that.
If someone else wants to come up with a proposal about improving the UX of this particular usecase together with an implementation sketch, then we can talk about the merits of that specific solution, and the maintenance burden that merging it will entail.

Spago is a package manager for PureScript, and as that it serves one particular customer with more attention than the rest: the Registry and its packages.
Handling git packages in the context of build plans is supported but not particularly pleasant (as you noticed), which is partly because of the reasons I laid out in my previous comment, and partly because it's supposed to nudge you to publish packages to the Registry.

In fact, I don't particularly follow why you're set on using git repos for everything: to follow the example packages you mentioned in the first post, dodo-printer is already in the registry, and the other one could be easily added.

If using git repos is particularly important for other reasons and you need to do this over several repos (as it is the case e.g. when using custom backends), then using a custom package set is another valid option.

[flip111]

it serves one particular customer with more attention than the rest: the Registry and its packages.

i definitely agree with this statement. IMO it's wonderful to have a cohesive package set that works together

then we can talk about the merits of that specific solution

ok that's cool !

I don't particularly follow why you're set on using git repos

sometimes major packages don't release new versions (for various reasons which are not interesting to get into IMO). I saw this several times. One example is this https://github.com/purescript-web/purescript-web-dom/commits/master/

The last release is from Apr 28, 2022 about 2.5 years ago. When i want to use commits from a year ago i can not use the package set from the registry.

it's supposed to nudge you to publish packages to the Registry

i understand the intent, sometimes people are busy or they don't want to publish for some reasons even when you ping them (that didn't happen in the given example, but on other occasions). It's just how it is i guess.

[f-f]

I definitely agree with this statement. IMO it's wonderful to have a cohesive package set that works together

Package sets are convenient, but that's not what I mean when I refer to registry packages: you can use packages from the registry without using a package set at all, and indeed that's the recommendation if you're developing a library.

The last release is from Apr 28, 2022 about 2.5 years ago. When i want to use commits from a year ago i can not use the package set from the registry.

I can cut a release for that. Would you help me with filling the rest of the changelog?

[flip111]

Great! Here is the updated changelog purescript-web/purescript-web-dom#60

What about adding a spago.yaml to that package?