[Feature] Package loader for amber

[b1ek]

we would obviously need a package manager at some point.

i'd like to propose a system for loading packages on compiler level

with these package directories (i will refer to this as $PKGDIR from now on):

1. ./amber_modules
2. ~/.local/amber_modules
3. /usr/lib/amber_modules
4. if AMBER_PATH is set, special logic applies

when importing a path that starts with a letter, compiler should import it from $PKGDIR/package_name/main.ab, with this logic (pseudocode):

// pseudo code
const package_directories = [
    "./amber_modules",
    "~/.local/amber_modules",
    "/usr/bin/amber_modules"
];

if (AMBER_PATH.is_set()) {
    const paths = AMBER_PATH.split(':');
    for (path of paths) {
        path = path.join('amber_modules');
        if (path.is_dir()) {
            # append from the top
            package_directories.push_front(path)
        }
    }
}

function resolve_import(path: string): string? {
    if !path.startsWithLetter() {
        return resolve_local_import(path)
    }
    if path.split("/").length == 1 {
        path += "/main.ab";
    }
    if !path.endsWith(".ab") {
        path += ".ab";
    }

    for (pkgdir of package_directories) {
        if file_exists(path.join(pkgdir)) {
            return path.join(pkgdir);
        }
        if exists(path.join(pkgdir) + "/main.ab") {
            return path.join(pkgdir) + "/main.ab";
        }
    }
    
    return null;
}

resolve_import("package_name"); // import "package_name" -> "$PKGDIR/main.ab"
resolve_import("package_name/subpkg"); // import "package_name" -> "$PKGDIR/subpkg.ab" | "$PKGDIR/subpkg/main.ab"
resolve_import("package_name/subpkg.ab"); // import "package_name" -> "$PKGDIR/subpkg.ab" | "$PKGDIR/subpkg/main.ab"

for importing files in the working directory, users should use import "./local_file"

implementing this will allow for creating a package manager for amber, that would be maintained independently from compiler

this will also mean treating stdlib as a package, rather than as a hardcoded language feature

AMBER_PATH logic

AMBER_PATH is an environment variable that behaves like PATH: it contains paths separated by :. for each path, amber must check if path/amber_modules exists and if it does, add it to potential package directories with high priority.

[lens0021]

I think $PKGDIR/main.ab is not enough, $PKGDIR/THE_SPECIFIED_VERSION_OF_THE_PACKAGE/main.ab is. So some counterparts of Cargo.toml and Cargo.lock are also required. So we should also decide to which format to use. (json, yaml, toml, pkl, kdl, etc).

And Jura is what I tried just for fun. But I will patch to match some of the requirements.

[b1ek]

I think $PKGDIR/main.ab is not enough, $PKGDIR/THE_SPECIFIED_VERSION_OF_THE_PACKAGE/main.ab is.

i wouldn't want to meddle with package versions, dependencies, and integrity checks on compiler level. i imagine it working like it is with node - a 3rd party app like yarn does all the smart work like downloading, versions and integrity checks, and puts it into node_modules for node to load. so like the compiler itself doesn't manage anything

also i think you have it a little bit confused - rust's compiler doesn't read cargo files. cargo is a totally independent program from rust's compiler

[b1ek]

i want to clarify that the purpose of this issue is not to create a package manager, but to create an interface for loading packages into the compiler. package management is not something that should be done by the compiler, but rather by a 3rd party program

[lens0021]

Oh, I am happy adding more location to search modules. Thank you.

My opinions for details.

• I am not sure if it is good to automatically load /main.ab when only package_name given. Because if both $PKGDIR/subpkg.ab and $PKGDIR/subpkg/main.ab exist, we should set priorities to them.
• Shouldn't importing a path that starts with a letter be for the project root? Though we have not the concept of the project root, I think we should if we treat packages.

[b1ek]

I am not sure if it is good to automatically load /main.ab when only package_name given. Because if both $PKGDIR/subpkg.ab and $PKGDIR/subpkg/main.ab exist, we should set priorities to them.

you mean $PKGDIR/package_name(.ab|/main.ab), right? if it is, then i wrote in the pseudocode snippet that the $PKGDIR/package_name/main.ab should be loaded only, and not $PKGDIR/package_name.ab, with the $PKGDIR/package_name being the namespace for that package

Shouldn't importing a path that starts with a letter be for the project root? Though we have not the concept of the project root, I think we should if we treat packages.

we do not have project root at all for now so i dont know how thats relevant. you do mean working directory, right?

i just feel like that would cause way too much confusion about what is a package and what is a file in the working directory

[b1ek]

also i think that in files that are imported in a different directory than the first one, working directory should refer to the directory they are in. so if ./main.ab imports ./dir/main, and ./dir/main.ab imports ./file, ./file should refer to ./dir/file.ab in that context. and if ./dir/main.ab needs to import something from the parent directory, it should use ../file_in_parent

[lens0021]

Thank you for the detailed explanation. I support your proposal.

[Thesola10]

Fundamentally Amber doesn't have a notion of a "project directory". Each file is imported relative to its importer, which is clear and easy to understand. Having to work around this leads to what Jura currently does to work around the lack of a configurable import path.

I still strongly believe an AMBER_PATH variable is the simpler way to address this, as showcased in #681, since the source code is not aware of the developer's environment and assuming so will break portability with Nix or Meson, or just with users in general.

There will be default paths, and the variable will only be used as an override as needed by a build tool or non-standard user config. Making any further assumptions will hinder Amber's adoption, as I believe embedding it into a project built in a different language will be a big use case.

If we want to perform version checks, or siloed environments, we can use external tools (or turn amber itself into a wrapper for, say, amberc). That way, the environment variable can be used to build virtual environments as needed, combining "default" modules (like the stdlib, or modules installed through rpm/pacman/what have you) with "local" modules in a predictable manner.