Describe the bug
The official Zephyr documentation for ztest framework claims that parameterized test cases are supported through usage of ZTEST_P macro.
This is factually not correct and leads to hours of wasted time when anybody tries to use it for parameterizing test cases.
What it actually does is (ab)using the already preexisted test suite fixtures functionality, but poorly and sideways.
The macro and the according documentation changes were introduced as part of #75457 . Author used a widely known and common term (parametrized tests) for something that in my opinion is only their own and very peculiar use case: "using Zephyr shell to execute a test case with a parameter".
There's a few problems with how this was accomplished:
- It creates confusion when somebody tries to use ZTEST_P macro, which is mentioned in ztest framework documentation, and not in ztest shell interface documentation. Users would try and use parameters lists or ranges in test suite code, but won't find how to do that - because there is no way, and the macro is doing something completely different from what documentation implies.
- When used, it breaks and replaces the preexisting test suite fixture functionality. If the same test suite tries to use a fixture - for example malloc a structure in setup(), use it in test cases and free it in tearDown() - the fixture will be overwritten by the "parameter" passed from a shell call when it happens, after setup() and before tearDown(), corrupting it.
- It is trivial to do what the original author needed using preexisting functionality - test suite fixture, or plain and simple static variables, with additional custom shell commands defined for a given test suite.
Steps to reproduce
- Read the ztest documentation.
- Try to use ZTEST_P for defining parameterized tests - that is, test cases with parameters lists or ranges to test on.
- Find out that it is not possible and that the documentation lies. Despair.
Impact
Annoyance – Minor irritation; no significant impact on usability or functionality.
Environment
- OS: any
- Environment: any
- Revision: latest main at the time of writing (81e1e71)
Describe the bug
The official Zephyr documentation for ztest framework claims that parameterized test cases are supported through usage of ZTEST_P macro.
This is factually not correct and leads to hours of wasted time when anybody tries to use it for parameterizing test cases.
What it actually does is (ab)using the already preexisted test suite fixtures functionality, but poorly and sideways.
The macro and the according documentation changes were introduced as part of #75457 . Author used a widely known and common term (parametrized tests) for something that in my opinion is only their own and very peculiar use case: "using Zephyr shell to execute a test case with a parameter".
There's a few problems with how this was accomplished:
Steps to reproduce
Impact
Annoyance – Minor irritation; no significant impact on usability or functionality.
Environment