run-vmtest: re-run denylisted tests for test_progs-bpf_gcc

The GCC BPF denylist is large and static, so tests stay on it long after the
compiler has learnt to build them. Nothing in CI notices, because denylisted
tests are never run.

After the main test_progs-bpf_gcc pass, run a second pass over the tests the
denylist excluded, by passing DENYLIST.test_progs-bpf_gcc back in as an
allowlist. Tests reported as passing there are candidates for removal from
the denylist.

Only the runner specific list is re-run, not the merged denylist the main
pass uses. Entries in the generic lists are excluded because they are broken
or unstable in the VM rather than because of the compiler, so re-running them
would risk taking the VM down without saying anything about GCC. The list is
consumed as-is: test_progs parses list files itself, skipping blank lines and
'#' comments, so it does not need to be normalized first.

The second pass is informational. Its exit code is deliberately not appended
to the exit status file, so tests that are still failing, which is the
expected outcome for most of them, cannot fail the job. No JSON summary is
written either: run.sh feeds every test_*.json to print_test_summary.py, which
truncates the step summary for each one, so a second file there would hide the
results of the main pass.

Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>

Author: vineetgarc

run-vmtest/run-bpf-selftests.sh

@@ -24,6 +24,7 @@ export BPFTOOL=$(find $(realpath "$SELFTESTS_BPF/tools/sbin") -type f -name bpft
 
 STATUS_FILE=${STATUS_FILE:-/mnt/vmtest/exitstatus}
 OUTPUT_DIR=${OUTPUT_DIR:-/mnt/vmtest}
+VMTEST_CONFIGS_DIR=${VMTEST_CONFIGS_DIR:-/mnt/vmtest/ci/vmtest/configs}
 
 test_progs_helper() {
   local selftest="test_progs${1}"
@@ -90,8 +91,41 @@ test_verifier() {
   foldable end test_verifier
 }
 
+# Re-run the tests that the denylist kept out of the main pass, by feeding
+# the denylist back in as an allowlist. Denylisted tests are expected to
+# fail, so this reports the ones that have started passing and can be
+# dropped from the list.
+#
+# Only the runner specific list is re-run, not the merged denylist the main
+# pass uses: entries in the generic lists are excluded because they are
+# broken or unstable in the VM, so re-running them risks taking the whole
+# machine down, and they say nothing about the compiler either way.
+#
+# This is informational: the exit code is deliberately not written to
+# ${STATUS_FILE}, so tests that are still failing cannot fail the job.
+test_progs_denylisted_helper() {
+  local selftest="test_progs${1}"
+  local denylist="${VMTEST_CONFIGS_DIR}/DENYLIST.${selftest}"
+  local args=()
+
+  if [ ! -s "${denylist}" ]; then
+    echo "${denylist} is missing or empty, nothing to re-test"
+    return 0
+  fi
+
+  args+=(${TEST_PROGS_WATCHDOG_TIMEOUT:+-w$TEST_PROGS_WATCHDOG_TIMEOUT})
+  args+=(-a@"${denylist}")
+
+  foldable start ${selftest}_denylisted "Re-testing denylisted ${selftest}"
+  echo "./${selftest}" "${args[@]}"
+  ./${selftest} "${args[@]}" && true
+  echo "${selftest} denylisted re-run exited with $?"
+  foldable end ${selftest}_denylisted
+}
+
 test_progs-bpf_gcc() {
     test_progs_helper "-bpf_gcc" ""
+    test_progs_denylisted_helper "-bpf_gcc"
 }
 
 export VERISTAT_TARGET=${VERISTAT_TARGET:-kernel}