Skip to content

Commit 89ee9d0

Browse files
authored
OPCT-304: cmd: adm parser-junit (#123)
Introduce a command to parse the JUnit files produced by e2e tests. This is a helper when running `openshift-tests` manually, it will provide quickly access to the results without struggling with parsing the XML file with `xq`. ~~~ $ opct-devel adm parse-junit failed_tests_rerun/results/junit_e2e__20240827-213043.xml Summary: - File: failed_tests_rerun/results/junit_e2e__20240827-213043.xml - Total: 19 - Pass: 9 - Skipped: 0 - Failures: 10 JUnit Attributes: - XMLName: { testsuite} - Name: openshift-tests - Tests: 19 - Skipped: 0 - Failures: 10 - Time: 2568 - Property: {TestVersion 4.14.0-202310201027.p0.g948001a.assembly.stream-948001a} #> Passed tests (9): [sig-instrumentation] MetricsGrabber should grab all metrics slis from API server. [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-auth][Feature:UserAPI] users can manipulate groups [apigroup:user.openshift.io][apigroup:authorization.openshift.io][apigroup:project.openshift.io] [Suite:openshift/conformance/parallel] [sig-cli] oc adm release extract image-references [Suite:openshift/conformance/parallel] [sig-auth][Feature:OAuthServer] OAuth server has the correct token and certificate fallback semantics [apigroup:user.openshift.io] [Suite:openshift/conformance/parallel] [sig-api-machinery][Feature:APIServer] anonymous browsers should get a 403 from / [Suite:openshift/conformance/parallel] [sig-auth][Feature:OAuthServer] well-known endpoint should be reachable [apigroup:route.openshift.io] [apigroup:oauth.openshift.io] [Suite:openshift/conformance/parallel] [sig-cli] oc basics can get version information from API [Suite:openshift/conformance/parallel] [sig-arch] Managed cluster should only include cluster daemonsets that have maxUnavailable or maxSurge update of 10 percent or maxUnavailable of 33 percent [Suite:openshift/conformance/parallel] [sig-arch] External binary usage #> Failed tests (10): "[sig-instrumentation] Prometheus [apigroup:image.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]" "[sig-network] Networking should provide Internet connection for containers [Feature:Networking-IPv4] [Skipped:Disconnected] [Skipped:Proxy] [Skipped:azure] [Suite:openshift/conformance/parallel] [Suite:k8s]" "[sig-builds][Feature:Builds] oc new-app should succeed with a --name of 58 characters [apigroup:build.openshift.io] [Skipped:Disconnected] [Skipped:Proxy] [Suite:openshift/conformance/parallel]" "[sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [apigroup:build.openshift.io] [Skipped:Disconnected] [Skipped:Proxy] [Suite:openshift/conformance/parallel]" "[sig-auth][Feature:OpenShiftAuthorization][Serial] authorization TestAuthorizationResourceAccessReview should succeed [apigroup:authorization.openshift.io] [Suite:openshift/conformance/serial]" "[sig-imageregistry][Feature:ImageInfo] Image info should display information about images [apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]" "[sig-cli] oc builds new-build [apigroup:build.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]" "[sig-network] services when running openshift ipv4 cluster ensures external ip policy is configured correctly on the cluster [apigroup:config.openshift.io] [Serial] [Suite:openshift/conformance/serial]" "[sig-instrumentation][Late] OpenShift alerting rules [apigroup:image.openshift.io] should link to a valid URL if the runbook_url annotation is defined [Suite:openshift/conformance/parallel]" "[sig-arch] External binary usage" #> Skipped tests (0): ~~~ It's useful when asking partners to replay tests / run `openshift-tests` manually. Example: https://issues.redhat.com/browse/OPCT-304
1 parent 634696b commit 89ee9d0

File tree

8 files changed

+417
-0
lines changed

8 files changed

+417
-0
lines changed

docs/assets/output/_generate-outputs.sh

+1
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ OUTPUT_DIR=${OUTPUT_OVERRIDE:-${DOC_ASSET_OUTPUT}}
2929
./build/opct-linux-amd64 adm generate checks-docs --help > "${OUTPUT_DIR}"/opct-adm-generate-checks-docs.txt
3030
./build/opct-linux-amd64 adm parse-etcd-logs --help > "${OUTPUT_DIR}"/opct-adm-parse-etcd-logs.txt
3131
./build/opct-linux-amd64 adm parse-metrics --help > "${OUTPUT_DIR}"/opct-adm-parse-metrics.txt
32+
./build/opct-linux-amd64 adm parse-junit --help > "${OUTPUT_DIR}"/opct-adm-parse-junit.txt
3233
./build/opct-linux-amd64 adm setup-node --help > "${OUTPUT_DIR}"/opct-adm-setup-node.txt
3334

3435

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
Administrative commands.
2+
3+
Usage:
4+
opct adm [flags]
5+
opct adm [command]
6+
7+
Available Commands:
8+
baseline Administrative commands to manipulate baseline results.
9+
cleaner Utility to apply pre-defined patches to existing result archive.
10+
e2e-dedicated Administrative commands for e2e-dedicated node configuration
11+
generate Generate administrative commands
12+
parse-etcd-logs Parse ETCD logs.
13+
parse-metrics Process the metrics collected by OPCT and create a HTML report graph.
14+
15+
Flags:
16+
-h, --help help for adm
17+
18+
Global Flags:
19+
--kubeconfig string kubeconfig for target OpenShift cluster
20+
--log-level string logging level (default "info")
21+
22+
Use "opct adm [command] --help" for more information about a command.

docs/opct/adm/parse-junit.md

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# opct adm parse-etcd-logs
2+
3+
Extract the information from JUnit files to get insights from a test execution.
4+
5+
## Options
6+
7+
```txt
8+
--8<-- "docs/assets/output/opct-adm-parse-junit.txt"
9+
```
10+
11+
## Usage
12+
13+
!!! info "Added to OPCT in release: v0.6+"
14+
15+
- Command: `opct adm parse-junit <path-to-junit-file>`
16+
17+
Args:
18+
19+
- `<path-to-junit-file>`: the path to the JUnit file.
20+
21+
## Examples
22+
23+
- Create the test list:
24+
25+
```sh
26+
cat << EOF >./test-list.txt
27+
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance] [Suite:openshift/conformance/serial/minimal] [Suite:k8s]"
28+
....
29+
EOF
30+
```
31+
32+
- Run `openshift-tests` targetting the test list:
33+
34+
> More information how to use [`openshift-tests`](./../../guides/features/running-e2e.md)
35+
36+
```sh
37+
./openshift-tests run \
38+
-f test-list.txt \
39+
--monitor="etcd-log-analyzer" \
40+
--junit-dir=./results
41+
```
42+
43+
- Run the parser command:
44+
45+
```bash
46+
$ ./build/opct-linux-amd64 adm parse-junit ./results/junit_e2e__20250313-192600.xml
47+
Summary:
48+
- File: ./results/junit_e2e__20250313-192600.xml
49+
- Total: 26
50+
- Pass: 23
51+
- Skipped: 0
52+
- Failures: 3
53+
54+
JUnit Attributes:
55+
- XMLName: { testsuite}
56+
- Name: openshift-tests
57+
- Tests: 26
58+
- Skipped: 0
59+
- Failures: 3
60+
- Time: 1557
61+
- Property: {TestVersion 4.19.0-202502102307.p0.gc22aad1.assembly.stream.el9-c22aad1}
62+
63+
#> Passed tests (23):
64+
[sig-api-machinery] Watchers should observe an object deletion if it stops meeting the requirements of the selector [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
65+
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeAffinity is respected if not matching [Suite:openshift/conformance/serial] [Suite:k8s]
66+
[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance] [Suite:openshift/conformance/serial/minimal] [Suite:k8s]
67+
[sig-instrumentation] MetricsGrabber should grab all metrics slis from API server. [Suite:openshift/conformance/parallel] [Suite:k8s]
68+
[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance] [Suite:openshift/conformance/serial/minimal] [Suite:k8s]
69+
[sig-scheduling] SchedulerPredicates [Serial] validates pod overhead is considered along with resource limits of pods that are allowed to run verify pod overhead is accounted for [Suite:openshift/conformance/serial] [Suite:k8s]
70+
[sig-cli] Kubectl client Kubectl diff should check if kubectl diff finds a difference for Deployments [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
71+
[sig-network][Feature:Router] The HAProxy router should enable openshift-monitoring to pull metrics [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
72+
[sig-network][Feature:Router] The HAProxy router should expose prometheus metrics for a route [apigroup:route.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
73+
[sig-auth][Feature:OAuthServer] OAuth server has the correct token and certificate fallback semantics [apigroup:user.openshift.io] [Suite:openshift/conformance/parallel]
74+
[sig-auth][Feature:OpenShiftAuthorization][Serial] authorization TestAuthorizationResourceAccessReview should succeed [apigroup:authorization.openshift.io] [Suite:openshift/conformance/serial]
75+
[sig-cli] oc builds new-build [apigroup:build.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
76+
[sig-network][Feature:Router] The HAProxy router should expose the profiling endpoints [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
77+
[sig-api-machinery][Feature:APIServer] anonymous browsers should get a 403 from / [Suite:openshift/conformance/parallel]
78+
[sig-arch] Managed cluster should ensure platform components have system-* priority class associated [Suite:openshift/conformance/parallel]
79+
[sig-node][apigroup:config.openshift.io] CPU Partitioning cluster infrastructure should be configured correctly [Suite:openshift/conformance/parallel]
80+
[sig-network][Feature:Router] The HAProxy router should expose a health check on the metrics port [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
81+
[sig-builds][Feature:Builds] oc new-app should succeed with an imagestream [apigroup:build.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
82+
[sig-auth][Feature:UserAPI] users can manipulate groups [apigroup:user.openshift.io][apigroup:authorization.openshift.io][apigroup:project.openshift.io] [Suite:openshift/conformance/parallel]
83+
[sig-cli] oc basics can get version information from API [Suite:openshift/conformance/parallel]
84+
[sig-auth][Feature:OAuthServer] well-known endpoint should be reachable [apigroup:route.openshift.io] [apigroup:oauth.openshift.io] [Suite:openshift/conformance/parallel]
85+
Cluster should be stable after installation is complete
86+
Cluster should be stable before test is started
87+
88+
#> Failed tests (3):
89+
"[sig-instrumentation] Prometheus [apigroup:image.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]"
90+
"[sig-network] services when running openshift ipv4 cluster ensures external ip policy is configured correctly on the cluster [apigroup:config.openshift.io] [Serial] [Suite:openshift/conformance/serial]"
91+
"[sig-network][Feature:tap] should create a pod with a tap interface [apigroup:k8s.cni.cncf.io] [Suite:openshift/conformance/parallel]"
92+
93+
#> Skipped tests (0):
94+
```

mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ nav:
5858
- generate checks-docs: opct/adm/generate-checks-docs.md
5959
- parse-etcd-logs: opct/adm/parse-etcd-logs.md
6060
- parse-metrics: opct/adm/parse-metrics.md
61+
- parse-junit: opct/adm/parse-junit.md
6162
- cleaner: opct/adm/cleaner.md
6263
- baseline: opct/adm/baseline.md
6364
- version: opct/version.md

pkg/api/junit.go

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
package api
2+
3+
import (
4+
"encoding/xml"
5+
"fmt"
6+
"os"
7+
8+
log "github.com/sirupsen/logrus"
9+
)
10+
11+
// Parse the XML data (JUnit created by openshift-tests)
12+
type TestStatus string
13+
14+
const (
15+
TestStatusPass TestStatus = "pass"
16+
TestStatusFail TestStatus = "fail"
17+
TestStatusSkipped TestStatus = "skipped"
18+
)
19+
20+
type propSkipped struct {
21+
Message string `xml:"message,attr"`
22+
}
23+
24+
type Property struct {
25+
Name string `xml:"name,attr"`
26+
Value string `xml:"value,attr"`
27+
}
28+
29+
type TestCase struct {
30+
Name string `xml:"name,attr"`
31+
Time string `xml:"time,attr"`
32+
Failure string `xml:"failure"`
33+
Skipped propSkipped `xml:"skipped"`
34+
SystemOut string `xml:"system-out"`
35+
Status TestStatus
36+
}
37+
38+
type TestSuite struct {
39+
XMLName xml.Name `xml:"testsuite"`
40+
Name string `xml:"name,attr"`
41+
Tests int `xml:"tests,attr"`
42+
Skipped int `xml:"skipped,attr"`
43+
Failures int `xml:"failures,attr"`
44+
Time string `xml:"time,attr"`
45+
Property Property `xml:"property"`
46+
Properties Property `xml:"properties,omitempty"`
47+
TestCases []TestCase `xml:"testcase"`
48+
}
49+
50+
type TestSuites struct {
51+
Tests int `xml:"tests,attr"`
52+
Disabled int `xml:"disabled,attr"`
53+
Errors int `xml:"errors,attr"`
54+
Failures int `xml:"failures,attr"`
55+
Time string `xml:"time,attr"`
56+
TestSuite TestSuite `xml:"testsuite"`
57+
}
58+
59+
type JUnitCounter struct {
60+
Total int
61+
Skipped int
62+
Failures int
63+
Pass int
64+
}
65+
66+
type JUnitXMLParser struct {
67+
XMLFile string
68+
Parsed *TestSuite
69+
Counters *JUnitCounter
70+
Failures []string
71+
Cases []*TestCase
72+
}
73+
74+
func NewJUnitXMLParser(xmlFile string) (*JUnitXMLParser, error) {
75+
p := &JUnitXMLParser{
76+
XMLFile: xmlFile,
77+
Parsed: &TestSuite{},
78+
Counters: &JUnitCounter{},
79+
Cases: []*TestCase{},
80+
}
81+
xmlData, err := os.ReadFile(xmlFile)
82+
if err != nil {
83+
return nil, fmt.Errorf("error reading XML file: %w", err)
84+
}
85+
if err := xml.Unmarshal(xmlData, p.Parsed); err != nil {
86+
ts := &TestSuites{}
87+
if err.Error() == "expected element type <testsuite> but have <testsuites>" {
88+
log.Warnf("Found errors while processing default JUnit format, attempting new JUnit format for e2e...")
89+
if err := xml.Unmarshal(xmlData, ts); err != nil {
90+
return nil, fmt.Errorf("error parsing XML data with testsuites: %w", err)
91+
}
92+
p.Parsed = &ts.TestSuite
93+
} else {
94+
return nil, fmt.Errorf("error parsing XML data: %w", err)
95+
}
96+
}
97+
// Iterate over the test cases
98+
for _, testcase := range p.Parsed.TestCases {
99+
// Create a new local copy of the testcase
100+
tc := &testcase
101+
p.Counters.Total++
102+
if len(testcase.Skipped.Message) > 0 {
103+
p.Counters.Skipped++
104+
tc.Status = TestStatusSkipped
105+
p.Cases = append(p.Cases, tc)
106+
continue
107+
}
108+
if len(testcase.Failure) > 0 {
109+
p.Counters.Failures++
110+
p.Failures = append(p.Failures, fmt.Sprintf("\"%s\"", testcase.Name))
111+
tc.Status = TestStatusFail
112+
p.Cases = append(p.Cases, tc)
113+
continue
114+
}
115+
tc.Status = TestStatusPass
116+
p.Cases = append(p.Cases, tc)
117+
}
118+
p.Counters.Pass = p.Counters.Total - (p.Counters.Skipped + p.Counters.Failures)
119+
120+
return p, nil
121+
}

pkg/api/junit_test.go

+96
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package api
2+
3+
import (
4+
"os"
5+
"testing"
6+
7+
"github.com/stretchr/testify/assert"
8+
)
9+
10+
func TestNewJUnitXMLParser(t *testing.T) {
11+
// Create a fake JUnit XML file
12+
xmlFile := createFakeJUnitXMLFileForOpenShiftTests()
13+
defer removeFakeJUnitXMLFile(xmlFile)
14+
15+
parser, err := NewJUnitXMLParser(xmlFile)
16+
assert.NoError(t, err)
17+
assert.NotNil(t, parser)
18+
19+
// Assert the parsed test suite
20+
assert.Equal(t, "openshift-tests", parser.Parsed.Name)
21+
assert.Equal(t, 3, parser.Parsed.Tests)
22+
assert.Equal(t, 1, parser.Parsed.Skipped)
23+
assert.Equal(t, 1, parser.Parsed.Failures)
24+
assert.Equal(t, "TestVersion", parser.Parsed.Property.Name)
25+
assert.Equal(t, "4.14.0-202310201027.p0.g948001a.assembly.stream-948001a", parser.Parsed.Property.Value)
26+
27+
// Assert the parsed test cases
28+
assert.Len(t, parser.Cases, 3)
29+
30+
// Assert the pass test case
31+
assert.Equal(t, "test_case_name_1", parser.Cases[0].Name)
32+
assert.Equal(t, "test_case_time_1", parser.Cases[0].Time)
33+
assert.Equal(t, "test_case_system_out_1", parser.Cases[0].SystemOut)
34+
assert.Equal(t, TestStatusPass, parser.Cases[0].Status)
35+
36+
// Assert the skipped test case
37+
assert.Equal(t, "test_case_name_2", parser.Cases[1].Name)
38+
assert.Equal(t, "test_case_time_2", parser.Cases[1].Time)
39+
assert.Equal(t, "test_case_system_out_2", parser.Cases[1].SystemOut)
40+
assert.Equal(t, TestStatusSkipped, parser.Cases[1].Status)
41+
assert.Equal(t, "test_case_skipped_message_2", parser.Cases[1].Skipped.Message)
42+
43+
// Assert the failed test case
44+
assert.Equal(t, "test_case_name_3", parser.Cases[2].Name)
45+
assert.Equal(t, "test_case_time_3", parser.Cases[2].Time)
46+
assert.Equal(t, "test_case_system_out_3", parser.Cases[2].SystemOut)
47+
assert.Equal(t, TestStatusFail, parser.Cases[2].Status)
48+
assert.Equal(t, "test_case_failure_3", parser.Cases[2].Failure)
49+
50+
// Assert the counters
51+
assert.Equal(t, 3, parser.Counters.Total)
52+
assert.Equal(t, 1, parser.Counters.Skipped)
53+
assert.Equal(t, 1, parser.Counters.Failures)
54+
assert.Equal(t, 1, parser.Counters.Pass)
55+
}
56+
57+
func createFakeJUnitXMLFileForOpenShiftTests() string {
58+
// Create a temporary file
59+
file, err := os.CreateTemp("", "opct-e2e.xml")
60+
if err != nil {
61+
panic(err)
62+
}
63+
defer file.Close()
64+
65+
// Case 1: Write the fake JUnit XML content to the file
66+
content := `<?xml version="1.0" encoding="UTF-8"?>
67+
<testsuite name="openshift-tests" tests="3" skipped="1" failures="1" time="2568">
68+
<property name="TestVersion" value="4.14.0-202310201027.p0.g948001a.assembly.stream-948001a"></property>
69+
<testcase name="test_case_name_1" time="test_case_time_1">
70+
<system-out>test_case_system_out_1</system-out>
71+
</testcase>
72+
<testcase name="test_case_name_2" time="test_case_time_2">
73+
<system-out>test_case_system_out_2</system-out>
74+
<skipped message="test_case_skipped_message_2"/>
75+
</testcase>
76+
<testcase name="test_case_name_3" time="test_case_time_3">
77+
<system-out>test_case_system_out_3</system-out>
78+
<failure>test_case_failure_3</failure>
79+
</testcase>
80+
</testsuite>`
81+
_, err = file.WriteString(content)
82+
if err != nil {
83+
panic(err)
84+
}
85+
86+
// Return the file path
87+
return file.Name()
88+
}
89+
90+
func removeFakeJUnitXMLFile(file string) {
91+
// Remove the temporary file
92+
err := os.Remove(file)
93+
if err != nil {
94+
panic(err)
95+
}
96+
}

0 commit comments

Comments
 (0)