What flavor tests do we want?

[philips]

From @wking on September 7, 2016 9:57
#257 shows we could really use a test suite. I'm flexible, but it feels like our command line API is more stable than our current Go APIs. How would folks feel about a test harness exercising the command line API? Do we have a sharness/bats preference (I have more experience with sharness, but runC uses bats)? Should this wait until after opencontainers/tob#18 (is there an estimated timeframe for that decision)?

Copied from original issue: opencontainers/image-spec#272

[philips]

From @glestaris on September 7, 2016 13:24

I agree that the CLI should be tested instead of any Go API. I would prefer if we could use a Go testing framework, even though it's a CLI. Ginkgo [1] is being used in Cloud Foundry for all types of testing (API/CLI/unit).

Also, I guess this issue is mainly targeting a CLI test suite, but, would it make sense to consider having unit tests on some of the most complex operations?

[1] http://onsi.github.io/ginkgo/

[philips]

From @wking on September 7, 2016 15:49

On Wed, Sep 07, 2016 at 06:24:42AM -0700, George Lestaris wrote:

I would prefer if we could use a Go testing framework, even though
it's a CLI. Ginkgo [1] is being used in Cloud Foundry for all types
of testing (API/CLI/unit).

Ginko looks nice, but I prefer shell-oriented test harnesses for the
command line because there's less boilerplate around invocation.
However, any framework around the command line is better than none, so
if maintainers want tests in Ginko (or whatever), I'll add them in
that framework.

Also, I guess this issue is mainly targeting a CLI test suite, but,
would it make sense to consider having unit tests on some of the
most complex operations?

As things currently stand, I think ‘oci-image-tool unpack SRC DEST’ is
more likely to be around in two months than manifest.unpack(walker,
dest), and testing I'd rather test the more stable interface. Testing
the command line API is also going to cover
cmd/oci-image-tool/unpack.go, and I'd rather have coverage first. We
can fill in unit tests later to make it easier to isolate errors and
avoid combinatoric issues ;). But again, if maintainers want Go tests
in image/*_test.go first, I'll go that way.