DAOS-19025 test: enospace.py functional tests is broken with new log … by knard38 · Pull Request #18391 · daos-stack/daos

knard38 · 2026-06-01T12:14:40Z

Description

Fixes NvmeEnospace functional tests (nvme/enospace.py) which all fail with "Expected DER_NOSPACE (-1007) should be > 0: got=0" on any CI run against a build that includes the new DAOS log format.

PR #17654 (DAOS-17989, merged 2026-05-11) moved the message severity level to the front of every log line:
Old format:

10/01-12:00:00.123456 ERR  DAOS[pid/tid] subsys file:line func() DER_NOSPACE(-1007): ...

New format:

ERR  2026/05/23 20:16:52.849308 hostname DAOS[pid/tid] subsys file:line func() DER_NOSPACE(-1007): ...

The sed match pattern in get_errors_count() (src/tests/ftest/util/general_utils.py) was written for the old format and requires at least one character before ERR:

^.+[[:space:]]ERR[[:space:]]...

With the new format, ERR is the first field on the line, so the pattern never matches. get_errors_count() returns an empty count, causing every verify_enospace_log() check to fail even though DER_NOSPACE errors are present in the log.

The proposed fix, update the match pattern to accept ERR at either the start of the line (new format) or preceded by a timestamp (old format), keeping backward compatibility:

-cmd += r"'/^.+[[:space:]]ERR[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"
+cmd += r"'/^(ERR|.+[[:space:]]ERR)[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"

The extraction sub-pattern (which captures the numeric error code) is unchanged and already works with both formats since DER_NOSPACE(-1007) always appears after some content on the line.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-06-01T12:15:02Z

Ticket title is 'enospace.py functional tests is broken with new log format'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-19025

daosbuild3 · 2026-06-01T15:00:30Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18391/1/execution/node/742/log

daosbuild3 · 2026-06-02T10:33:43Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18391/2/execution/node/738/log

daosbuild3 · 2026-06-03T20:24:22Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18391/5/execution/node/743/log

daosbuild3 · 2026-06-05T06:58:41Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18391/6/testReport/

…format PR #17654 (DAOS-17989) moved the severity level to the front of each log line. The sed match pattern in get_errors_count() requires at least one character before ERR, so it never matches the new format and all NvmeEnospace subtests fail with DER_NOSPACE count=0. Fix: update the pattern to also accept ERR at the start of the line: ^(ERR|.+[[:space:]]ERR)[[:space:]]... Quick-Functional: true Test-tag: NvmeEnospace Test-repeat: 5 Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

daosbuild3 · 2026-06-06T08:38:26Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18391/7/testReport/

grom72 · 2026-06-08T06:29:41Z

    # Get the Client side Error from client_log file.
    cmd = "cat {} | sed -n -E -e ".format(get_log_file(file_glob))
-    cmd += r"'/^.+[[:space:]]ERR[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"
+    cmd += r"'/^(ERR|.+[[:space:]]ERR)[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"


Backward compatibility is not required.
I don't know of any situation where we use the master branch test version with release (2.6/2.8) RPMs.

Suggested change

cmd += r"'/^(ERR|.+[[:space:]]ERR)[[:space:]].+[[:space:]]DER_[^(]+$[^)]+$.+$/"

cmd += r"'/^ERR|[[:space:]].+[[:space:]]DER_[^(]+$[^)]+$.+$/"

There is technically one case: the interoperability tests :)
But those tests do not call this code

Backward compatibility is not required. I don't know of any situation where we use the master branch test version with release (2.6/2.8) RPMs.

Remove backward compatibility

Fixed with commit 5dad11d

…/daos-19025/patch-001

…format Fix reviewer comments: - remove backward compatibility Quick-Functional: true Test-tag: NvmeEnospace Test-repeat: 5 Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

…/daos-19025/patch-001 Quick-Functional: true Test-tag: NvmeEnospace Test-repeat: 5

daosbuild3 · 2026-06-10T15:21:09Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18391/9/execution/node/787/log

daosbuild3 · 2026-06-10T15:59:40Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18391/9/execution/node/872/log

daosbuild3 · 2026-06-11T01:14:12Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18391/9/testReport/

knard38 · 2026-06-12T13:07:41Z

The failure of the last build #9 is a known issue reported in the ticket DAOS-19033.

knard38 · 2026-06-12T13:19:18Z

@daos-stack/daos-gatekeeper could you lend this PR with the following message:

title: DAOS-19025 test: fix enospace log parsing for new log format
body:

PR #17654 moved the severity level to the front of log lines. The sed pattern
in get_errors_count() required at least one character before ERR, so it never
matched. Update the pattern to accept ERR at line start.

knard38 self-assigned this Jun 1, 2026

knard38 force-pushed the ckochhof/fix/master/daos-19025/patch-001 branch from df382ca to 8399dbf Compare June 3, 2026 08:03

knard38 force-pushed the ckochhof/fix/master/daos-19025/patch-001 branch from 8399dbf to 3ee04f8 Compare June 4, 2026 14:22

knard38 force-pushed the ckochhof/fix/master/daos-19025/patch-001 branch from 3ee04f8 to 9ed0e9f Compare June 5, 2026 14:52

knard38 marked this pull request as ready for review June 5, 2026 14:54

knard38 requested review from a team as code owners June 5, 2026 14:54

knard38 requested review from daltonbohning and grom72 June 5, 2026 14:56

daltonbohning previously approved these changes Jun 5, 2026

View reviewed changes

grom72 requested changes Jun 8, 2026

View reviewed changes

kanard38 added 2 commits June 9, 2026 12:49

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

9a5d8a6

…/daos-19025/patch-001

DAOS-19025 test: enospace.py functional tests is broken with new log …

5dad11d

…format Fix reviewer comments: - remove backward compatibility Quick-Functional: true Test-tag: NvmeEnospace Test-repeat: 5 Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>

knard38 dismissed daltonbohning’s stale review via 5dad11d June 9, 2026 12:57

knard38 requested review from daltonbohning and grom72 June 9, 2026 12:59

grom72 approved these changes Jun 9, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into ckochhof/fix/master…

cfdf8bd

…/daos-19025/patch-001 Quick-Functional: true Test-tag: NvmeEnospace Test-repeat: 5

daltonbohning approved these changes Jun 10, 2026

View reviewed changes

knard38 added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jun 12, 2026

knard38 requested a review from a team June 12, 2026 13:21

daltonbohning merged commit 1643e26 into master Jun 12, 2026
33 of 38 checks passed

daltonbohning deleted the ckochhof/fix/master/daos-19025/patch-001 branch June 12, 2026 17:26

	cmd += r"'/^(ERR\|.+[[:space:]]ERR)[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"
	cmd += r"'/^ERR\|[[:space:]].+[[:space:]]DER_[^(]+\([^)]+\).+$/"

Conversation

knard38 commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jun 1, 2026

Uh oh!

daosbuild3 commented Jun 2, 2026

Uh oh!

daosbuild3 commented Jun 3, 2026

Uh oh!

daosbuild3 commented Jun 5, 2026

Uh oh!

daosbuild3 commented Jun 6, 2026

Uh oh!

grom72 Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

daltonbohning Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jun 10, 2026

Uh oh!

daosbuild3 commented Jun 10, 2026

Uh oh!

daosbuild3 commented Jun 11, 2026

Uh oh!

knard38 commented Jun 12, 2026

Uh oh!

knard38 commented Jun 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

knard38 commented Jun 1, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading

knard38 Jun 9, 2026 •

edited

Loading