Test coverage

[ahx]

Here test coverage means that the gem should be able to track and report which requests and reponses of an API description have been called during a test run.
The more request and response definitions have been called the better.

This would unaware about JSON Schema and it's branches like if/else, oneOf, anyOf, type: […] or the like, but only care about paths, response statuses and content-types.

Configuration could work like this:

# in spec_helper.rb
OpenapiFirst::Coverage.register('my/openapi.yaml', only: ->(path) { path.start_with?('/pets') })

RSpec.configure do |_config|
  after(:suite) do
    # Generates a HTML report in 'coverage/openapi_first/index.html'
    # Returns coverage data
    # raises 'Not all paths have been called' if not all paths have been called
    # … or just one thing at a time?
    OpenapiFirst::Coverage.report
  end
end

Then you just run your tests and it will track all calls RuntimeRequest#validate and RuntimeResponse#validate. So this will just work either if you are using the built in middlewares (which you should) or roll your own.

Questions:

1. OpenapiFirst::Test::Coverage or OpenapiFirst::Coverage?
2. Extra gem or this gem? Maybe html formatter in extra gem similar to simplecov

[MrBananaLord]

That's a nice idea!

# raises 'Not all paths have been called' if not all paths have been called
# … or just one thing at a time?

Maybe raise an error optionally like: OpenapiFirst::Coverage.report(raise_error: true)

Extra gem or this gem? Maybe html formatter in extra gem similar to simplecov

Do you want to keep the Coverage in this gem and have a separate gem for building the HTML for it? Sounds ok to me, but if the formatting part is small it might be just fine to keep it here as well. Kind of unsure about that :)

[ahx]

I am working on adding test coverage and this is how it looks so far. I am wondering if it should complain about untested 4xx, 5xx responses by default if they are described for the request or just skip them. 🤔

[ahx]

Hey. There is "coverage" branch with something working, finally. Check it out. Any feedback on this would be greatly appreciated: https://github.com/ahx/openapi_first/tree/coverage?tab=readme-ov-file#test-coverage

TODOs:

• Fail (exit 1) if coverage is not 100%
• Optionally skip 401,403 responses?