Add proposal for unified results directory structure#79
Add proposal for unified results directory structure#79maryamtahhan wants to merge 2 commits intoredhat-et:mainfrom
Conversation
This proposal addresses inconsistent results directories across test types and adds support for custom test run names as requested in issue redhat-et#73. Key changes: - Create docs/proposals/ directory for design proposals - Add results-directory-structure.md with detailed proposal - Update docs/docs.md to reference new proposal The proposal includes: - Unified 6-level hierarchy for LLM tests (type/suite/model/workload/run/config) - Simplified 5-level hierarchy for embedding tests (type/scenario/model/run/config) - Custom test run names via parameter or environment variable - Migration strategy with 4 non-breaking phases - Streamlit visualization support for filtering by test run name Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
📝 WalkthroughWalkthroughIntroduces a new design proposals documentation system for the vLLM CPU Performance Evaluation framework. Establishes a proposals hub with standardized workflow stages and templates, followed by the first proposal defining a unified results directory structure for LLM and embedding test outputs with naming conventions, validation rules, and a phased migration strategy. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Add language identifiers to code blocks to prevent comment lines from being treated as H1 headings (MD025) - Restart numbered lists at 1 for each subsection (MD029) - Convert comment-style annotations to plain text descriptions Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Maryam Tahhan <mtahhan@redhat.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/docs.md`:
- Line 133: Update the stale date in the documentation table row that currently
reads "proposals/results-directory-structure.md | 🚧 In Review | 2024-03-30" to
the correct creation date "2026-03-30"; locate the string in docs/docs.md (the
table entry for proposals/results-directory-structure.md) and replace the year
portion of the date only so the row becomes
"proposals/results-directory-structure.md | 🚧 In Review | 2026-03-30".
In `@docs/proposals/results-directory-structure.md`:
- Around line 396-399: The checklist links in results-directory-structure.md
point one level too shallow; update each relative link in the checklist (the
entries referencing automation/test-execution/ansible/ansible.md, README.md,
results/results.md, and the migration guide line) to use the correct path from
docs/proposals (prefix with ../../) so they resolve to the repository root
(e.g., change ../automation/... to ../../automation/..., ../README.md to
../../README.md, and ../results/results.md to ../../results/results.md).
- Around line 106-112: Remove the stray opening code fence before the "### Path
Examples" heading that is breaking the markdown structure; locate the "### Path
Examples" and the subsequent "#### LLM Examples" section and delete the
unintended ``` so the headings render normally, and verify that the intended
fenced blocks (e.g., the ```text``` that follows "#### LLM Examples") are
properly balanced and only surround the intended example content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 85930f4a-5bbd-44ff-be15-a6a18f0cf467
📒 Files selected for processing (3)
docs/docs.mddocs/proposals/README.mddocs/proposals/results-directory-structure.md
| | methodology/metrics.md | ✅ Complete | 2024-02-08 | | ||
| | methodology/reporting.md | ✅ Complete | 2024-02-08 | | ||
| | platform-setup/x86/intel/deterministic-benchmarking.md | ✅ Complete | (current) | | ||
| | proposals/results-directory-structure.md | 🚧 In Review | 2024-03-30 | |
There was a problem hiding this comment.
Update stale “Last Updated” year for the new proposal entry.
Line 133 uses 2024-03-30, but this PR was created on 2026-03-30. Keeping the year aligned avoids timeline confusion in the status table.
Suggested fix
-| proposals/results-directory-structure.md | 🚧 In Review | 2024-03-30 |
+| proposals/results-directory-structure.md | 🚧 In Review | 2026-03-30 |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | proposals/results-directory-structure.md | 🚧 In Review | 2024-03-30 | | |
| | proposals/results-directory-structure.md | 🚧 In Review | 2026-03-30 | |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/docs.md` at line 133, Update the stale date in the documentation table
row that currently reads "proposals/results-directory-structure.md | 🚧 In
Review | 2024-03-30" to the correct creation date "2026-03-30"; locate the
string in docs/docs.md (the table entry for
proposals/results-directory-structure.md) and replace the year portion of the
date only so the row becomes "proposals/results-directory-structure.md | 🚧 In
Review | 2026-03-30".
| ``` | ||
|
|
||
| ### Path Examples | ||
|
|
||
| #### LLM Examples | ||
|
|
||
| ```text |
There was a problem hiding this comment.
Remove stray code fence that breaks markdown structure.
Line 106 opens an unintended fenced block; this can swallow subsequent headings/content and render the document incorrectly.
Suggested fix
#### Embedding Tests (5 levels)
1. **Model Type** (`embedding/`) - Top-level categorization
2. **Scenario** - Test scenario (baseline, latency, concurrent)
3. **Model** - Specific model being tested
4. **Run Name** - Unique identifier (custom or timestamp)
5. **Config** - Configuration (if applicable - often just files at run_name level)
-```
### Path Examples🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/proposals/results-directory-structure.md` around lines 106 - 112, Remove
the stray opening code fence before the "### Path Examples" heading that is
breaking the markdown structure; locate the "### Path Examples" and the
subsequent "#### LLM Examples" section and delete the unintended ``` so the
headings render normally, and verify that the intended fenced blocks (e.g., the
```text``` that follows "#### LLM Examples") are properly balanced and only
surround the intended example content.
| - [ ] Update [automation/test-execution/ansible/ansible.md](../automation/test-execution/ansible/ansible.md) | ||
| - [ ] Update [README.md](../README.md) | ||
| - [ ] Update [results/results.md](../results/results.md) (if exists) | ||
| - [ ] Add migration guide |
There was a problem hiding this comment.
Fix relative links in checklist (currently one level short).
From docs/proposals/results-directory-structure.md, links to repo-root docs should use ../../..., otherwise navigation breaks.
Suggested fix
-- [ ] Update [automation/test-execution/ansible/ansible.md](../automation/test-execution/ansible/ansible.md)
-- [ ] Update [README.md](../README.md)
-- [ ] Update [results/results.md](../results/results.md) (if exists)
+- [ ] Update [automation/test-execution/ansible/ansible.md](../../automation/test-execution/ansible/ansible.md)
+- [ ] Update [README.md](../../README.md)
+- [ ] Update [results/results.md](../../results/results.md) (if exists)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/proposals/results-directory-structure.md` around lines 396 - 399, The
checklist links in results-directory-structure.md point one level too shallow;
update each relative link in the checklist (the entries referencing
automation/test-execution/ansible/ansible.md, README.md, results/results.md, and
the migration guide line) to use the correct path from docs/proposals (prefix
with ../../) so they resolve to the repository root (e.g., change
../automation/... to ../../automation/..., ../README.md to ../../README.md, and
../results/results.md to ../../results/results.md).
|
Providing users to specify names for test-runs as an environment variable is key. After the testing is complete these (and other) specific results should be selectable in the StreamLIT Dashboard. Ideally allowing a Comparative View to visualize multiple 'custom names' in a single Graph with selectable Metrics in a pull-down menu. |
2 participants
This proposal addresses inconsistent results directories across test types and adds support for custom test run names as requested in issue #73.
Key changes:
The proposal includes:
Summary by CodeRabbit