What's the canonical way to do "argument forwarding" (wrapping exec)?

[enkore]

This does pop up from time to time and the answer seems to be generally "trailing_var_arg + allow_hyphen_values", but this has various unexpected failure modes. Summary of the tests below:

Regular positional arg (e.g. binary name) followed by var_arg (e.g. options for the binary) fails like this:

1. "--help" and similar are actually picked up inside the var arg. Seems like a bug to me.
2. Regular options are some times ignored (-xfoo), sometimes picked up (-x foo). Seems like a bug to me.

Single var_arg for everything fails like this

1. Short options are not parsed, -x foo is parsed as an option, while -xfoo starts the var_arg. Seems like a bug to me.

Single var_arg with last = true appears to be the only option that reliably works, but has the following drawbacks:

1. Requires use of --, even though the resulting parser isn't generally ambigiuous under getopt/getopt_long rules (any positional terminates parsing of options and flags).
2. Precludes splitting the argument list into two parts

This use case seems common enough that there should be one preferred and documented way to do it which does not fail in strange ways. Most of the odd behavior here seems to come from clap's decision to allow mixing positional arguments with options and flags (much like argparse does, though clap seems to have fewer bugs than argparse). I've looked around and haven't seen anything to disable this (ime undesirable) behavior.

use clap::Parser;

fn main() {
    // clap v4.4.18
    // clap_derive v4.4.7

    #[derive(Parser, Debug)]
    pub struct ExecCommand {
        #[arg(long, short = 'x')]
        some_option: Option<String>,

        /// Command to run
        #[arg(required = true)]
        exec: String,

        /// Arguments to the command
        #[arg(required = false, trailing_var_arg = true, allow_hyphen_values = true)]
        args: Vec<String>,
    }

    //println!("{:#?}", ExecCommand::parse_from(["test", "/usr/bin/cp", "-h"]));
    // prints:
    // Usage: test [OPTIONS] <EXEC> [ARGS]...
    //
    // Arguments:
    //   <EXEC>     Command to run
    //   [ARGS]...  Arguments to the command
    //
    // Options:
    //   -v, --verbose
    //   -x, --some-option <SOME_OPTION>
    //   -h, --help

    println!("{:#?}", ExecCommand::parse_from(["test", "-xfoo", "/usr/bin/cp", "arg1", "arg2"]));
    // ExecCommand {
    //     some_option: Some(
    //         "foo",
    //     ),
    //     exec: "/usr/bin/cp",
    //     args: [
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    println!("{:#?}", ExecCommand::parse_from(["test", "/usr/bin/cp", "-xfoo", "arg1", "arg2"]));
    // Weirdly enough this works:
    // ExecCommand {
    //     some_option: None,
    //     exec: "/usr/bin/cp",
    //     args: [
    //         "-xfoo",  // <- option ignored
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    println!("{:#?}", ExecCommand::parse_from(["test", "/usr/bin/cp", "-x", "foo", "arg1", "arg2"]));
    // ... unless you use a space.
    // ExecCommand {
    //     some_option: Some( // <- whoopsie
    //         "foo",
    //     ),
    //     exec: "/usr/bin/cp",
    //     args: [
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    #[derive(Parser, Debug)]
    pub struct ExecCommandOneArg {
        #[arg(long, short = 'x')]
        some_option: Option<String>,

        /// Command with arguments
        #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)]
        args: Vec<String>,
    }

    println!("{:#?}", ExecCommandOneArg::parse_from(["test", "/usr/bin/cp", "-h"]));
    // ExecCommandOneArg {
    //     some_option: None,
    //     args: [
    //         "/usr/bin/cp",
    //         "-h",
    //     ],
    // }

    // but watch this
    println!("{:#?}", ExecCommandOneArg::parse_from(["test", "-xfoo", "/usr/bin/cp", "arg1", "arg2"]));
    // ExecCommandOneArg {
    //     some_option: None,
    //     args: [
    //         "-xfoo",     // <-- whoops!
    //         "/usr/bin/cp",
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    println!("{:#?}", ExecCommandOneArg::parse_from(["test", "-x", "foo", "/usr/bin/cp", "arg1", "arg2"]));
    // must suddenly use spaced form
    // ExecCommandOneArg {
    //     some_option: Some(
    //         "foo",
    //     ),
    //     args: [
    //         "/usr/bin/cp",
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    #[derive(Parser, Debug)]
    pub struct ExecCommandLast {
        #[arg(long, short = 'x')]
        some_option: Option<String>,

        /// Command to run
        #[arg(required = true, last = true)]
        exec: String,

        /// Arguments to the command
        #[arg(required = false, trailing_var_arg = true, allow_hyphen_values = true)]
        args: Vec<String>,
    }

    // println!("{:#?}", ExecCommandLast::parse_from(["test", "-xfoo", "/usr/bin/cp", "arg1", "arg2"]));
    // error: unexpected argument '/usr/bin/cp' found
    //
    // Usage: test [OPTIONS] -- <EXEC> [ARGS]...
    //
    // For more information, try '--help'.
    //
    // Fair enough, the docs say this, though I would've expected this to mean something more similar
    // getopt-style parsing, where the first argument with this flag terminates parsing of options/flags

    // println!("{:#?}", ExecCommandLast::parse_from(["test", "-xfoo", "--", "/usr/bin/cp", "arg1", "arg2"]));
    // Except it doesn't seem to work:
    // error: the following required arguments were not provided:
    //   <EXEC>
    //
    // Usage: test --some-option <SOME_OPTION> -- <EXEC> <ARGS>...
    //
    // For more information, try '--help'.

    // println!("{:#?}", ExecCommandLast::parse_from(["test", "--", "/usr/bin/cp", "arg1", "arg2"]));
    // nope
    // error: the following required arguments were not provided:
    //   <EXEC>
    //
    // Usage: test -- <EXEC> <ARGS>...
    //
    // For more information, try '--help'.

    // I think the problem here is that last=true cannot be followed by another argument, so in
    // clap's model "exec" is now the physically last argument, but since it's then preceded by
    // "args", "args" sucks up all the arguments and "exec" gets nothing.


    #[derive(Parser, Debug)]
    pub struct ExecCommandOneArgLast {
        #[arg(long, short = 'x')]
        some_option: Option<String>,

        /// Command to run
        #[arg(required = true, last = true)]
        args: Vec<String>,
    }

    // println!("{:#?}", ExecCommandOneArgLast::parse_from(["test", "-xfoo", "/usr/bin/cp", "arg1", "arg2"]));
    // error: unexpected argument '/usr/bin/cp' found
    //
    // Usage: test [OPTIONS] -- <ARGS>...
    //
    // For more information, try '--help'.

    println!("{:#?}", ExecCommandOneArgLast::parse_from(["test", "-xfoo", "--", "/usr/bin/cp", "arg1", "arg2"]));
    // ExecCommandOneArgLast {
    //     some_option: Some(
    //         "foo",
    //     ),
    //     args: [
    //         "/usr/bin/cp",
    //         "arg1",
    //         "arg2",
    //     ],
    // }

    println!("{:#?}", ExecCommandOneArgLast::parse_from(["test", "--", "/usr/bin/cp", "-h"]));
    // ExecCommandOneArgLast {
    //     some_option: None,
    //     args: [
    //         "/usr/bin/cp",
    //         "-h",
    //     ],
    // }
}

[indirect]

We're running into the same issue over here: spinel-coop/rv#426.

It seems like it should be possible to tell clap we want [OPTIONS] COMMAND [ARGS], allow OPTIONS to contain flags that are parsed by clap, and allow ARGS to contain flags that are not parsed by clap. We're planning to work around it by combining COMMAND and [ARGS] into a single Vec that clap does not parse, and then separate out COMMAND manually, but that makes setting up the help much more awkward.

Why can't we set COMMAND as required, and then capture flags in ARGS the exact same way as the combined cmd_args captures them?

[epage]

Why can't we set COMMAND as required, and then capture flags in ARGS the exact same way as the combined cmd_args captures them?

Sorry, I can't think of a better answer than "clap isn't set up that way". There are a lot of different use cases and a lot of different ways to design a library to meet those needs. The fact that we can meet those needs with cmd_args is sufficient for us. We could add more flags to meet very specific use cases but then everyone else pays for the build time, binary size, and runtime of that customization.

Note: I'd recommend removing trailing_var_args(true) from cmd_args. That only describes how to start the argument and not what gets captured later.

[indirect]

Thanks for the explanation. It's possible I missed something, but I spent around an hour reading the docs and trying different combinations of num_args and required and trailing_var_args thinking I was just holding it wrong before finally giving up and finding many tickets and discussions around the same problem.

I think it would help tremendously for the clap documentation to explicitly mention your recommendation, since the various issues and discussions make it seem fairly common to want some form of "one required argument followed by fully captured trailing arguments".

[epage]

Sorry, this was posted during a very busy time for me and I apparently haven't made it this far back into my backlog.

There are multiple use cases and we need to separate them out

• argument forwarding, generally trailing_var_arg(true).allow_hyphen_values(true)
• command and argument forwarding, either external_subcommand(true) or trailing_var_arg(true).

So that would be:

    #[derive(Parser, Debug)]
    pub struct ExecCommandOneArg {
        #[arg(long, short = 'x')]
        some_option: Option<String>,

        /// Command with arguments
        #[arg(required = true, trailing_var_arg = true)]
        args: Vec<String>,
    }

[epage]

As for the confusion over -xfoo and -x foo. this is because of #3410. When we evaluate whether a short is a short or a hyphen value, we are not taking attached values into account.