Is your feature request related to a problem? Please describe.
When an e2e test fails, all of the e2e tests are re-run because the retry action we use will just run make e2e-gh-token again.
This causes a bunch of extra work to be done (token quota and time to run)
Describe the solution you'd like
ginkgo supports a --flake-attempts argument, which will only re-run the specs that fail.
ginkgo help run
--flake-attempts [int] (default: 0 - failed tests are not retried)
Make up to this many attempts to run each spec. If any of the attempts
succeed, the suite will not be failed.
If we add --flake-attempts=3 to the right spots in the Makefile, our e2e tests might look something like this:
• [FAILED] [3.364 seconds]
E2E TEST:Code-Review E2E TEST:Validating use of code reviews [It] Should return inconclusive results for a single-maintainer project with only self- or bot changesets
~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:105
Captured StdOut/StdErr Output >>
2023/04/21 15:25:24 use code reviews: utests.TestReturn{
... // 2 identical fields
NumberOfWarn: 0,
NumberOfInfo: 0,
- NumberOfDebug: 0,
+ NumberOfDebug: 1,
}
2023/04/21 15:25:25 use code reviews: utests.TestReturn{
... // 2 identical fields
NumberOfWarn: 0,
NumberOfInfo: 0,
- NumberOfDebug: 0,
+ NumberOfDebug: 1,
}
2023/04/21 15:25:26 use code reviews: utests.TestReturn{
... // 2 identical fields
NumberOfWarn: 0,
NumberOfInfo: 0,
- NumberOfDebug: 0,
+ NumberOfDebug: 1,
}
<< Captured StdOut/StdErr Output
Timeline >>
[FAILED] in [It] - ~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:124 @ 04/21/23 15:25:24.482
Attempt #1 Failed. Retrying ↺ @ 04/21/23 15:25:24.483
[FAILED] in [It] - ~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:124 @ 04/21/23 15:25:25.512
Attempt #2 Failed. Retrying ↺ @ 04/21/23 15:25:25.512
[FAILED] in [It] - ~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:124 @ 04/21/23 15:25:26.589
<< Timeline
[FAILED] Expected
<bool>: false
to be true
In [It] at: ~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:124 @ 04/21/23 15:25:26.589
There were additional failures detected. To view them in detail run ginkgo -vv
Summarizing 1 Failure:
[FAIL] E2E TEST:Code-Review E2E TEST:Validating use of code reviews [It] Should return inconclusive results for a single-maintainer project with only self- or bot changesets
~/go/src/github.com/spencerschrock/scorecard.git/e2e/code_review_test.go:124
Ran 54 of 57 Specs in 197.156 seconds
FAIL! -- 53 Passed | 1 Failed | 0 Pending | 3 Skipped
Is your feature request related to a problem? Please describe.
When an e2e test fails, all of the e2e tests are re-run because the retry action we use will just run
make e2e-gh-tokenagain.This causes a bunch of extra work to be done (token quota and time to run)
Describe the solution you'd like
ginkgo supports a
--flake-attemptsargument, which will only re-run the specs that fail.ginkgo help runIf we add
--flake-attempts=3to the right spots in the Makefile, our e2e tests might look something like this: