Support ignoring/skipping tests at runtime
As it stands, when protocol is unavailable the test fails. This is non-optimal from a regression testing standpoint.
- If the test goes from "fail but really should skip" to "fail" we need to know.
- We want CI to be green even if there are skips.
There are a few options...
Filter the output
Sudbury used a post processing script to filter out fail/skip https://gitlab.freedesktop.org/bwidawsk/sudbury/-/blob/8fcb53052fd26e29cbf82c53c26e0ada5f0d606d/.gitlab-ci.yml#L153. Something similar can be done based on fail reason (I believe), but this also requires a more consumable output format.
Don't add tests that can't pass
The original plan was to have the test harness (tests/tests/main.rs) enumerate the globals and only add tests that will have enough available. I think this won't work well because globals can appear and disappear dynamically.
Dynamically ignore tests based on instantaneous runtime information
I think the right solution is to have the test harness be able to ignore/skip itself at test run time. There was already a feature request to add this to libtest-mimic.e