Coverage reporting missing for binaries under-test

[epage]

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

[epage]

See also the discussion at assert-rs/assert_cli#104

[glehmann]

I'm exactly in this situation right now, and unfortunately, I have no idea how to fix it.
I've tried kcov, gcov and tarpaulin without success so far. I'm afraid I may have to go back to testing my cli tool directly by calling directly the main function from my crate.
I really like the idea to test the final executable, in order to make sure to not miss anything, and assert_cmd makes is quite pleasant to do, so this is a bit disappointing!

[glehmann]

I've been able to get kcov to report something by calling kcov instead of calling directly my program.

https://github.com/glehmann/hld/blob/f40f3b51f84969cc13714f81451ad17f33fbf2dc/tests/common/mod.rs#L79

I activate it with the kcov feature in my project. kcov doesn't seem to be able to run in parallel, so I force the test to run on a single thread:

cargo test --features kcov -- --test-threads 1

It would be nice to have something similar natively, for example with an environment variable that allow the developer to set a wrapper — in that case, kcov. Something like

RUST_ASSERT_CMD_WRAPPER="kcov --include-pattern=/src --exclude-pattern=/.cargo $PWD/cov" cargo test  -- --test-threads 1

[epage]

A wrapper sounds like a great way to resolve this since the coverage tools don't follow fork (granted, when running cargo would we want them to)

[epage]

I'm assuming I'll need to use shlex to separate out the command from the args when calling Command::new

[maxcountryman]

I’ll add that this would be a very nice feature. I’m currently using assert_cmd and tarpaulin and just like others have noted that this combination doesn’t work.

[brandonkal]

grcov (and kcov) appears to work with assert_cmd. Other tools failed to mark lines as tested as others have mentioned here.

[pkgw]

My project doesn't use assert_cmd, but based on my investigations of kcov I am pretty sure that special support will need to be added to properly track coverage in subcommand executions. You can check out tectonic-typesetting/tectonic#638 to see my approach:

• Gather the tests that do assert_cmd-type activities into one harness (called executable in my project)
• Run the test suite inside kcov, but not doing anything special when launching subcommands (I use cargo kcov which currently requires some hacking, but that's a separate issue)
• For the executable harness, rerun it outside of kcov, but with a special environment variable that causes it to launch the underlying tool inside kcov; it puts the coverage data into a mkdtemp-type directory name so that runs don't stop on each other's toes
• Merge all of the coverage results together at the end with kcov --merge

[frosklis]

Is there any plan on having this solved? Like @maxcountryman , I've seen that it doesn't work with tarpaulin coverage reports.