Package crowbar.0.2.2

[stedolan]

crowbar.0.2.2

Crowbar is a library for testing code, combining QuickCheck-style
property-based testing and the magical bug-finding powers of
afl-fuzz.

[jmid]

I'm seeing a number of CI test errors, however they seem to be happening on 4.x. not 5.x, e.g.:
https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/d420daa195894543660aadc8251d362e9076b201/variant/compilers,4.08,crowbar.0.2.2,tests

#=== ERROR while compiling crowbar.0.2.2 ======================================#
# context              2.5.0~alpha1 | linux/x86_64 | ocaml-base-compiler.4.08.1 | pinned(https://github.com/stedolan/crowbar/archive/v0.2.2.tar.gz)
# path                 ~/.opam/4.08/.opam-switch/build/crowbar.0.2.2
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p crowbar -j 71 --promote-install-files=false @install @runtest
# exit-code            1
# env-file             ~/.opam/log/crowbar-7-849f65.env
# output-file          ~/.opam/log/crowbar-7-849f65.out
### output ###
# (cd _build/default/examples/fpath && ./test_fpath.exe)
# segs: PASS
# 
# (cd _build/default/examples/map && ./test_map.exe)
# map: PASS
# 
# (cd _build/default/examples/pprint && ./test_pprint.exe)
# pprint: PASS
# 
# (cd _build/default/examples/serializer && ./test_serializer.exe)
# pairs: PASS
# 
# (cd _build/default/examples/uunf && ./test_uunf.exe)
# uunf: PASS
# 
# File "examples/calendar/dune", line 2, characters 7-20:
# 2 |  (name test_calendar)
#            ^^^^^^^^^^^^^
# (cd _build/default/examples/calendar && ./test_calendar.exe)
# calendar: FAIL
# 
# When given the input:
# 
#     [1711-06-30 00:19:24; 1713-03-29 00:38:50]
#     
# the test failed:
# 
#     1713-04-01 00:38:50 != 1713-03-29 00:38:50
# 

One thing that changed from 4.x to 5.x was the PRNG. I wonder if that could explain 4.x test differences? 🤔

Unrelated: Would you consider adding an x-maintenance-intent entry?
https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md

[stedolan]

One thing that changed from 4.x to 5.x was the PRNG. I wonder if that could explain 4.x test differences? 🤔

Yes, this turns out to be it. The test is to run some crowbar tests of existing opam libraries, and PRNG differences mean that on OCaml 4 the test is now correctly detecting ocaml-community/calendar#13. I'm tempted to leave this as-is: the test hasn't changed since prior releases of crowbar, and the bug it's detecting is not a bug in this package. Does having packages with failing tests cause a problem? (In any case, I'll make the tests less janky for the next release).

Unrelated: Would you consider adding an x-maintenance-intent entry?
https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md

Sure, I'll try to find some time to read through that spec and add one.

[jmid]

The test is to run some crowbar tests of existing opam libraries, and PRNG differences mean that on OCaml 4 the test is now correctly detecting ocaml-community/calendar#13. I'm tempted to leave this as-is: the test hasn't changed since prior releases of crowbar, and the bug it's detecting is not a bug in this package. Does having packages with failing tests cause a problem?

The failing test will show up as a reverse-dependency red light for the next release of, e.g., dune or cmdliner. This adds noise and thus makes it harder to identify actual regressions.

It is really great that crowbar is finding bugs! For the purposes of the opam-repo CI, the package tests would preferably work (deterministically) across both OCaml 4.x and 5.x. I know this takes some effort to realize though. For QCheck, for example, I've ended up with 4 different expect test outputs ({ocaml4, ocaml5} x {32bit, 64bit}) because of PRNG variations 😬 For a quicker fix, one could disable Crowbar's runtest on OCaml 4.x entirely, or perhaps just the calendar test? 🤔

Unrelated: Would you consider adding an x-maintenance-intent entry?
https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md

Sure, I'll try to find some time to read through that spec and add one.

TLDR, you can add x-maintenance-intent: [ "(latest)" ] like the 98% other packages to indicate that you will only commit to maintaining the latest version. This lets opam maintainers archive old package versions, and thus save CPU cycles from retesting these. 🙏

[stedolan]

The failing test will show up as a reverse-dependency red light for the next release of, e.g., dune or cmdliner. This adds noise and thus makes it harder to identify actual regressions.

Makes sense! For now I've just disabled the calendar test entirely - it's just an example and it's not really testing anything not covered by the other examples.

TLDR, you can add x-maintenance-intent: [ "(latest)" ] like the 98% other packages to indicate that you will only commit to maintaining the latest version. This lets opam maintainers archive old package versions, and thus save CPU cycles from retesting these. 🙏

Done.

[jmid]

Thanks, this LGTM!

CI summary

• uunf.17.0.0 failed to build on flambda (a known issue)
• an opam-2.0 cyclic-dependency error (also a known issue)
• a macos howebrew infrastructure error ("switch turned off")
• a number of revdeps failures
  • a carton build failure (fixed in #28923)
  • decompress runtest compilation failures ("Error: Library "bigarray-compat" not found", "Error: Library "unix" was pulled in")
  • MlFront_{Cli,Core} runtest fuzzparsing failures
  • data-encoding runtest assertion failures

The latter two looks like they could be caused by a different fuzzer PRNG seeding, like the calendar-one. The failures cannot be blamed on this release however.