Evaluate program with dynamic information present in env

[wezm]

If I'm using Nickel as a library in Rust code how would I go about evaluating a program with a module/import consisting of dynamic information already present in the environment akin to how std is loaded in mk_eval_env?

For example if I had a configuration file like:

{
  interface = machine.iface.name,
  bind = machine.iface.ip,
  port = 80
}

When evaluated it would produce something like this (although it would be deserialised into Rust types instead of generating JSON):

{
  "iface": "eth0",
  "ip": "192.168.1.21",
  "port": 80
}

I'd like to be able to pro-populate machine with data read at runtime by the program using nickel-lang-core.

Alternatively if the is a better way to achieve the same thing I'd be happy to have suggestions.

[suimong]

One possibility would be:

1. Define your machines in a separate nickel file

# machines.ncl
{
  A = {
    iface.name = "eth0",
    iface.ip = "192.168.1.121",
  },
  B = {
    iface.name = "eth1",
    iface.ip = "172.16.0.5",
  },
}

2. In your actual config, declare "machine" as an input field

# config.ncl
{
  inputs = {
    machine,
  },
  outputs = {
    interface = inputs.machine.iface.name,
    bind = inputs.machine.iface.ip,
    port = 80,
  },
}

3. Utilize nickel CLI's customize-mode to "pro-populate" a machine of your choice.

nickel export config.ncl --field=outputs -- 'inputs.machine=(let machines = import "machines.ncl" in machines.A)'

[yannham]

In general, I think it's better to have an explicit dependency on the machine parameter, as in @suimong 's solution (or it could be a function parameter, although record fields are more flexible). It shouldn't be too hard to do the same without resorting to the CLI, if you're embedding Nickel in your Rust application: the logic of extracting one specific field and applying overrides is entirely implemented inside program::Program. The CLI is just a layer of text interface over that. See for example add_overrides and field_path that you can set directly.

That being said, if for your application some functions make sense as a custom standard library because they are really pervasive, then you'll have to use nickel_lang_core::eval::VirtualMachine directly, which is what Program is using under the hood. Unfortunately Program doesn't provide access to the virtual machine, so you'll have to handle some of the things that Program do manually; however, if your setting is simple (no overrides, no field to extract in particular), this shouldn't be too hard. I haven't tried this code in an actual workflow, but a first try should be something like:

 pub fn eval_in_env<T, S>(
        source: T,
        source_name: S,
        trace: impl Write + 'static,
        // Put your additional stuff in this environment
        env: Environment,
    ) -> Result<RichTerm, Error>
    where
        T: Read,
        S: Into<OsString> + Clone,
    {
        // Mostly Program::new_from_source
        let mut cache = Cache::new(ErrorTolerance::Strict);
        let path = PathBuf::from(source_name.into());
        // you probably don't want unwrap here
        let main_id = cache.add_source(SourcePath::Path(path), source).unwrap();
        let mut vm = VirtualMachine::new(cache, trace);

        // Mostly Program::prepare_eval() 
        let prepared = vm.prepare_eval(main_id)?;
        // If you want to evaluate deeply and substitute variables
        // Mostly Program::eval_full
        Ok(vm.eval_full_closure(Closure { body: prepared, env})?.body)
    }

However, there's a subtle issue here: the prepare_eval typechecks the program, and the typechecker will error with unbound variable errors (the symbols that you are adding to the library). While you can always descend deeper into the different modules and probably inline nice helper functions and change a bit their content, I fear this might become a bit tedious, albeit possible.

One simple work-around for now is to simply eschew typechecking altogether:

    pub fn eval_in_env<T, S>(
        source: T,
        source_name: S,
        trace: impl Write + 'static,
        extension: Environment,
    ) -> Result<RichTerm, Error>
    where
        T: Read,
        S: Into<OsString> + Clone,
    {
        // Mostly Program::new_from_source
        let mut cache = Cache::new(ErrorTolerance::Strict);
        let path = PathBuf::from(source_name.into());
        // you probably don't want unwrap here
        let main_id = cache.add_source(SourcePath::Path(path), source).unwrap();
        let mut vm = VirtualMachine::new(cache, trace);

        // Custom preparation phase: prepare does three steps. Parsing, typechecking and program
        // transformations. We just eschew the typechecking.
        vm.import_resolver_mut().parse(main_id).unwrap();
        vm.import_resolver_mut().transform(main_id).unwrap();
        let prepared = vm.import_resolver_mut().get_owned(main_id).unwrap();

        // If you want to evaluate deeply and substitute variables
        // Mostly Program::eval_full
        Ok(vm.eval_full_closure(Closure { body: prepared, env: extension})?.body)
    }

I think this is the simplest approach. Doing typechecking with a custom stdlib is technically possible, but more tedious, because the current infrastructure wasn't built with this use-case in mind. So the initial typing environment is mostly automatically generated from a hard-coded list of stdlib module. By using directly the cache (vm.import_resolver_mut()), you can decompose the preparation even further and inject your own type environment, but you'll have to build it by hand, so to speak (I'm thinking about the initial typecheck::Context to provide to typechecker::type_check)

[wezm]

Thank you both for the detailed replies and pointers. I'll take your suggestions and see what I can work out.