Skip to content

WIP: Rust score calculation #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

nicoburns
Copy link
Contributor

@nicoburns nicoburns commented Feb 18, 2025

THIS PR IS NOT INTENDED FOR MERGING IN IT'S CURRENT FORM

Actual score calculation is implemented in the wptreport crate:

This PR adds a CI job to run the CLI from the crate on Github CI here with the purpose of seeing how fast the code runs on GIthub Actions.

Todo:

  • Implement "ingesting a new report"
    • Merging WPT report fragments
    • Converting WPT report to "Servo Test Scores" format (stored in this repo)
    • Add a CLI command
    • xz compression
    • Add to CI job
  • Make focus areas configurable
  • Make output location configurable
  • Output scores-last-run.json (or remove)
  • Use cargo-binstall to eliminate wpt cli installation time
  • Use in nightly CI job

@nicoburns
Copy link
Contributor Author

Ok: this PR is doing score recalculation in 2m 49s vs. 27m 8s on main. That's a pretty nice improvement!

@mukilan
Copy link
Member

mukilan commented Feb 20, 2025

The numbers look really promising! I've given the crate only a cursory read through - it seems like if the goal is to use this for Servo's WPT dashboard, the pending items are

  • A way to recalculate past runs by scoring them against a given run.
  • Support regex based areas to support /css/CSS2/tables & /css/css-tables. Doesn't have to be regex, I guess, we just need a way to combine arbitrary folders.
  • Few unit tests for confidence. Maybe just generating scores.json against a fixed dataset using the crate and diffing it against the current one would also work.

@nicoburns
Copy link
Contributor Author

The numbers look really promising! I've given the crate only a cursory read through - it seems like if the goal is to use this for Servo's WPT dashboard, the pending items are

Some up to date numbers: this PR is now calculating in 4m 18s vs 35m 47s for the existing JS version.

  • A way to recalculate past runs by scoring them against a given run.

This is now implemented. Unfortunately it is a little slower, but it's not too bad.

  • Support regex based areas to support /css/CSS2/tables & /css/css-tables. Doesn't have to be regex, I guess, we just need a way to combine arbitrary folders.

This is implemented as an array of areas to support. Any exact prefix of segment (separated by a /) is supported (/css/, /css/CSS2 and /css/CSS2/tables will all work, but /css/CSS2/tab would not). Currently this is hardcoded into the CLI, but I will probably make it load from the JSON file.

  • Few unit tests for confidence. Maybe just generating scores.json against a fixed dataset using the crate and diffing it against the current one would also work.

I have generating identical output to #67 modulo float precision (in the last 2 decimal places of numbers with ~8 decimal places - so I expect this is fine?). This PR is generating a scores.json file and uploading it as an artefact if anyone wishes to compare

@nicoburns nicoburns force-pushed the rust-score-calculation branch 10 times, most recently from fdfc750 to 37f06e3 Compare July 19, 2025 00:26
@nicoburns nicoburns force-pushed the rust-score-calculation branch from 37f06e3 to f0fbfd7 Compare July 19, 2025 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants