Missing options: Available in RunLTP and not in Kirk

[Sowmya-Indranna]

As continued to explore more on the kirk available options, noticed that certain options available in runltp are not present in kirk and wanted to check if there are any alternatives or plans to provide equivalent options. Specifically, looking for these options:

1. These two options are available in runltp:
   a) -I ITERATIONS Execute the testsuite ITERATIONS times

          b)  -t DURATION     Execute the testsuite for given duration.
   

2. kirk with option: --workers WORKERS, -w WORKERS Number of workers to execute tests in parallel

Ran one of the ltp test suite syscalls with and without -w option:

./kirk -r syscalls

              results shows: **Execution time: 56m 6s**
                                     Suite Name: syscalls
                                     Total Run: 1458
                                     Total Runtime: 50m 51s
                                     Passed Tests: 15270
                                     Failed Tests: 30
                                     Skipped Tests: 638
                                     Broken Tests: 2
                                     Warnings: 39

./kirk -r syscalls -w 100

             results shows: **Execution time: 57m 58s**
                                       Suite Name: syscalls
                                       Total Run: 1458
                                       Total Runtime: 1h 2m 58s
                                       Passed Tests: 14102
                                       Failed Tests: 32
                                       Skipped Tests: 553
                                       Broken Tests: 4
                                       Warnings: 39

Is it limiting the parallel execution? Same goes with --force-parallel also -p option

[acerv]

• -t (for single tests execution) and -T (for session execution) are available in kirk as well. What's missing?
• -I has never been a priority, I don't really know if it's required @metan-ucw what do you think?
• it's possible that parallelization takes more time. You need to find a right threshold for the number of workers, otherwise you might saturate CPU and I/O, causing certain tests to take too much time to complete and slowing down the whole execution. There's not a smart way (yet) to prioritize certain tests instead of others, according to their characteristics. This might be possible once we add tags in the new LTP API, so we can decide how to mix them together during parallelization

[Sowmya-Indranna]

Rechecking with the timeout options available.

[Sowmya-Indranna]

testing with -w and observations:

Resources available:
root@dixilp10:~ free -h
total used free shared buff/cache available
Mem: 67Gi 11Gi 61Gi 16Mi 550Mi 56Gi
Swap: 13Gi 683Mi 13Gi
root@dixilp10:~
root@dixilp10: nproc
56

tested with minimal workers and execution looks good

root@dixilp10:~ ps -ef | grep 15763
root 15763 5100 2 00:09 hvc0 00:00:01 python3 ./kirk -r syscalls -w 2
root 29361 15763 0 00:10 ? 00:00:00 inotify09
root 29374 15763 0 00:10 ? 00:00:00 inotify11
root 29379 28605 0 00:10 pts/0 00:00:00 grep --color=auto 15763

root@dixilp10:~ ps -ef | grep 32225
root 32225 5100 6 00:11 hvc0 00:00:03 python3 ./kirk -r syscalls -w 5
root 45073 32225 0 00:12 ? 00:00:00 inotify09
root 50303 32225 0 00:12 ? 00:00:00 select02
root 51123 32225 0 00:12 ? 00:00:00 times03
root 51165 32225 0 00:12 ? 00:00:00 timerfd_settime02
root 51351 32225 0 00:12 ? 00:00:00 waitpid01
root 51481 28605 0 00:12 pts/0 00:00:00 grep --color=auto 32225

Increasing to 10 workers, didn't run as expected

root@dixilp10:/opt/ltp ps -ef | grep 54679
root 54679 5100 1 00:13 hvc0 00:00:03 python3 ./kirk -r syscalls -w 10
root 77279 54679 0 00:14 ? 00:00:00 bind06
root 77896 28605 0 00:18 pts/0 00:00:00 grep --color=auto 54679
root@dixilp10:/opt/ltp ps -ef | grep 54679
root 54679 5100 1 00:13 hvc0 00:00:04 python3 ./kirk -r syscalls -w 10
root 94744 54679 1 00:19 ? 00:00:00 fallocate05
root 94823 28605 0 00:19 pts/0 00:00:00 grep --color=auto 54679
root@dixilp10:/opt/ltp ps -ef | grep 54679
root 54679 5100 1 00:13 hvc0 00:00:04 python3 ./kirk -r syscalls -w 10
root 105982 54679 0 00:19 ? 00:00:00 fcntl36_64
root 106949 28605 0 00:19 pts/0 00:00:00 grep --color=auto 54679
root@dixilp10:/opt/ltp ps -ef | grep 54679
root 30383 28605 0 00:20 pts/0 00:00:00 grep --color=auto 54679
root 54679 5100 1 00:13 hvc0 00:00:04 python3 ./kirk -r syscalls -w 10
root 109230 54679 0 00:20 ? 00:00:00 fork13

availability of resources here during execution:
root@dixilp10:/opt/ltp free -h
total used free shared buff/cache available
Mem: 67Gi 13Gi 58Gi 29Mi 680Mi 53Gi
Swap: 14Gi 436Mi 13Gi

However using force-parallel -p options did pick number of workers:

root@dixilp10:/opt/ltp ps -ef | grep 9718
root 5803 9718 0 03:37 ? 00:00:00 mmap16
root 6264 9718 0 03:37 ? 00:00:00 mount03
root 9718 5100 9 03:37 hvc0 00:00:04 python3 ./kirk -r syscalls -w 10 -p
root 10597 9718 0 03:37 ? 00:00:00 mount06
root 11902 9718 0 03:37 ? 00:00:00 mount07
root 12190 28605 0 03:37 pts/0 00:00:00 grep --color=auto 9718
root 12721 9718 0 03:37 ? 00:00:00 bind06
root 13234 9718 0 03:37 ? 00:00:00 clock_settime03
root 20863 9718 0 03:37 ? 00:00:00 fork13
root 20883 9718 5 03:37 ? 00:00:00 landlock04
root 28716 9718 0 03:37 ? 00:00:00 kill02
root 32676 9718 0 03:37 ? 00:00:00 mkdir09
root@dixilp10:/opt/ltp ps -ef | grep 9718
root 6264 9718 0 03:37 ? 00:00:00 mount03
root 9718 5100 9 03:37 hvc0 00:00:06 python3 ./kirk -r syscalls -w 10 -p
root 12721 9718 0 03:37 ? 00:00:00 bind06
root 13234 9718 0 03:37 ? 00:00:00 clock_settime03
root 15217 9718 0 03:38 ? 00:00:00 poll02
root 15512 9718 0 03:38 ? 00:00:00 prctl09
root 16485 9718 3 03:38 ? 00:00:00 preadv03
root 16708 9718 3 03:38 ? 00:00:00 preadv03_64
root 17416 9718 0 03:38 ? 00:00:00 preadv203
root 18307 28605 0 03:38 pts/0 00:00:00 grep --color=auto 9718
root 20863 9718 0 03:37 ? 00:00:00 fork13
root 31145 9718 0 03:37 ? 00:00:00 nanosleep01

[chetjain]

• -t (for single tests execution) and -T (for session execution) are available in kirk as well. What's missing?

  • -I has never been a priority, I don't really know if it's required @metan-ucw what do you think?

  • it's possible that parallelization takes more time. You need to find a right threshold for the number of workers, otherwise you might saturate CPU and I/O, causing certain tests to take too much time to complete and slowing down the whole execution. There's not a smart way (yet) to prioritize certain tests instead of others, according to their characteristics. This might be possible once we add tags in the new LTP API, so we can decide how to mix them together during parallelization

Having -I is important v/s

• -t (for single tests execution) and -T (for session execution) are available in kirk as well. What's missing?

  • -I has never been a priority, I don't really know if it's required @metan-ucw what do you think?

The timeouts always stop the execution abruptly, whereas -I ends gracefully. -I also helps in bench-marking & POC to claim the number of times the tests attempted, whereas -t all depends on machine response & performance

[pevik]

@acerv Thanks for changing -S context to just runtest only.

The timeouts always stop the execution abruptly, whereas -I ends gracefully. -I also helps in bench-marking & POC to claim the number of times the tests attempted, whereas -t all depends on machine response & performance

I don't know who is really using -I, but it should not be that hard to take integer argument and pass it to each test right?

BTW if implemented, I would suggest to name it -i to match LTP -i argument. Also there is -I which specifies seconds instead of times. How about to implement them both? (-i must be integer <0, ? - probably INT_MAX>, -I has range <0.1, INT_MAX>)

-i n     Execute test n times
-I x     Execute test for n seconds

[Sowmya-Indranna]

@acerv Thanks for changing -S context to just runtest only.

The timeouts always stop the execution abruptly, whereas -I ends gracefully. -I also helps in bench-marking & POC to claim the number of times the tests attempted, whereas -t all depends on machine response & performance

I don't know who is really using -I, but it should not be that hard to take integer argument and pass it to each test right?

BTW if implemented, I would suggest to name it -i to match LTP -i argument. Also there is -I which specifies seconds instead of times. How about to implement them both? (-i must be integer <0, ? - probably INT_MAX>, -I has range <0.1, INT_MAX>)

-i n     Execute test n times
-I x     Execute test for n seconds

@pevik -i and -I are already in use in kirk
--skip-tests SKIP_TESTS, -i SKIP_TESTS Skip specific tests
--skip-file SKIP_FILE, -I SKIP FILE Skip specific tests using a skip file (newline separated item)

[pevik]

@pevik -i and -I are already in use in kirk
--skip-tests SKIP_TESTS, -i SKIP_TESTS Skip specific tests
--skip-file SKIP_FILE, -I SKIP FILE Skip specific tests using a skip file (newline separated item)

@Sowmya-Indranna thx!
@acerv @metan-ucw I would be for rename them, it's still quite early and hopefully not many frameworks support them yet. IMHO we should try to match LTP library getopts (fortunately only few) and runltp getopts (these we implement).

[acerv]

I don't think it's a big issue to implement -I option as well as changing the options names, but is it really needed to change them? At the end, as you change the command name, arguments will follow. What we are really missing is some features from kirk and that was somehow expected, since we never planned to have already all features if people don't really require them.

[Sowmya-Indranna]

I don't think it's a big issue to implement -I option as well as changing the options names, but is it really needed to change them? At the end, as you change the command name, arguments will follow. What we are really missing is some features from kirk and that was somehow expected, since we never planned to have already all features if people don't really require them.

We know some Kirk features were expected to be missing, but this option is something we actively use. Adding it would enhance usability without affecting existing functionality. Given our use case, we’d really appreciate its inclusion.

[Sowmya-Indranna]

kirk options: -T SUITE_TIMEOUT Timeout before stopping the suite - Working as expected:
./kirk -r syscalls -T 60
Host information
System: Linux
Node: devtest01
Kernel Release: X.X.X.X
Kernel Version: #1 SMP Mon Sep 30 10:17:22 EDT 2024
Machine Architecture: ppc64le
Processor: ppc64le
Temporary directory: /tmp/kirk.root/tmpk47nwl0v
Connecting to SUT: host
Starting suite: syscalls
abort01: pass | tainted (0.155s)
accept01: pass | tainted (0.006s)
accept02: pass | tainted (0.010s)
bpf_prog05: skip | tainted (16.793s)
bpf_prog06: skip | tainted (16.792s)
bpf_prog07: Suite 'syscalls' timed out after 60 seconds

Execution time: 1m 0s
Suite Name: syscalls

Point 1: whereas -t EXEC_TIMEOUT Timeout before stopping a single execution

./kirk -r syscalls -c accept01 -t 60 ## Executes single test once and continues with the suite execution
Host information
System: Linux
Node: devtest01
Kernel Release: X.X.X.X
Kernel Version: #1 SMP Mon Sep 30 10:17:22 EDT 2024
Machine Architecture: ppc64le
Processor: ppc64le
Temporary directory: /tmp/kirk.root/tmpd_nuln27
Connecting to SUT: host
accept01
tst_test.c:1900: TINFO: LTP version: 20250130-118-g0f9d817a3
tst_test.c:1904: TINFO: Tested kernel: X.X.X.X#1 SMP Mon Sep 30 10:17:22 EDT 2024 ppc64le
tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/X.X.X.X.ppc64le/build/.config'
tst_test.c:1722: TINFO: Overall timeout per run is 0h 00m 05s
accept01.c:91: TPASS: bad file descriptor : EBADF (9)
accept01.c:91: TPASS: invalid socket buffer : EINVAL (22)
accept01.c:91: TPASS: invalid salen : EINVAL (22)
accept01.c:91: TPASS: no queued connections : EINVAL (22)
accept01.c:91: TPASS: UDP accept : EOPNOTSUPP (95
Summary:
passed 5
failed 0
broken 0
skipped 0
warnings 0
Exit code: 0

Starting suite: syscalls
abort01: pass | tainted (0.122s)
accept01: pass | tainted (0.002s)
accept02: pass | tainted (0.005s)
accept03: skip | tainted (0.060s)
accept4_01: pass | tainted (0.005s)
add_key02: pass | tainted (0.002s)
add_key04: pass | tainted (0.003s)
adjtimex03: pass | tainted (0.003s)
alarm02: pass | tainted (0.002s)
alarm03: pass | tainted (0.002s)
......

In runltp, the single test executes continuously until the timeout - Shouldn't it work the same way?

./runltp -f syscalls -s accept01 -t 1s

Point 2: Also, -t DURATION n runltp, execute the testsuite for given duration. Examples:
-t 60s = 60 seconds
-t 45m = 45 minutes
-t 24h = 24 hours
-t 2d = 2 days
No unit validation is available here (m|s|h|d)

[acerv]

@Sowmya-Indranna now I see your point. runltp has a feature which was continuously running a test for a specific amount of time. kirk lacks this feature and it only provides a timeout. I guess this is the second feature we need to add after -I in order to get closer to runltp.

[metan-ucw]

@pevik I find it absurd that the skip short option isn't -s but -i that should be fixed.

As for the timeout -t and -T these shouldn't even exist, as we have all the data for timeouts in the metadata now and kirk should be smart enough to use that.

As for the ability to run tests in a loop for longer time that does sound like a useful option. I guess that we need something as:

• Ability to pass down -i and -I options to the testcases
• Option that would run random test from the lists of tests until we run out of allocated time

[Sowmya-Indranna]

@Sowmya-Indranna now I see your point. runltp has a feature which was continuously running a test for a specific amount of time. kirk lacks this feature and it only provides a timeout. I guess this is the second feature we need to add after -I in order to get closer to runltp.

yes, but with -t option, its not timing out but continues suite execution

[acerv]

@metan-ucw I'm the first who doesn't want to have -t and -T honestly. But we are still facing new and old API problem.