docs(submitqueue): combine predictor into Scorer - #682
Merged
Merged
Conversation
This was referenced Sep 7, 2026
behinddwalls
force-pushed
the
preetam/outcome-predictor-rfc
branch
from
September 7, 2026 23:54
451805e to
3227132
Compare
behinddwalls
marked this pull request as ready for review
September 8, 2026 00:08
behinddwalls
force-pushed
the
preetam/outcome-predictor-rfc
branch
from
September 8, 2026 00:57
01a9de2 to
9696e21
Compare
sbalabanov
requested changes
Sep 10, 2026
sbalabanov
left a comment
sbalabanov
left a comment
Contributor
There was a problem hiding this comment.
- Is predictor a first class abstraction (i.e. an interface) or implementation detail of the generator?
- Let's discuss as an alternative to have all the decisions inside scorer. Scorer is called for each batch anyways? Why can't be made evidence-aware?
- The proposed implementation of the predictor is somewhat hard to read. Can we frame it as a linear regression model with defined features and configured weights? What in the proposal does not fit into a definition of LGM?
behinddwalls
changed the title
Collaborator
Author
|
Agreed on all three — the stack now does (1)+(2) as one Scorer, and (3) is documented on that impl.
The rewrite is in this PR; impl / wiring / demo are #683, #684, #686. |
sbalabanov
approved these changes
Sep 12, 2026
## Summary ### Why? Speculation ranks paths on the scorer's price, which only reflects the change (lines, files) and ignores what the run has since observed — a passed build, a failed build, batch state. The predictor design needs a written contract before the implementation lands. ### What? Adds doc/rfc/submitqueue/outcome-predictor.md: scorer vs predictor, the four YAML factors and what their values mean, which path results count as evidence, and rejected alternatives. Links it from doc/rfc/index.md. ## Test Plan Docs-only change.
## Summary ### Why? The RFC's rejected-options list used implementation shorthand and included alternatives already explained elsewhere, making the design tradeoffs difficult to evaluate without branch context. ### What? Keep the four alternatives reviewers are likely to raise and state each rejected option, why it fails, and the selected design. Remove redundant and scorer-specific entries.
## Summary ### Why? The outcome predictor RFC described failed-path evidence and ranking differently from the path-set identity and best-first behavior, while older speculation RFCs still assigned dynamic pricing to the scorer. ### What? Document that prediction applies to unresolved dependencies, failed all-succeeds evidence applies once, neutral factors preserve exact scorer prices, and factors must be finite. Update the existing speculation RFCs to consistently describe predictor-based ranking and state-aware pricing. # Conflicts: # doc/rfc/submitqueue/speculation-generator-best-first.md # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 91ddcce # Last commands done (3 commands done): # pick 3227132 # docs(submitqueue): clarify rejected predictor alternatives # pick 7d17f2d # docs(submitqueue): reconcile predictor RFCs # Next commands to do (2 remaining commands): # pick 9696e21 # docs(submitqueue): complete outcome predictor RFC # pick f98b8e8 # docs(submitqueue): combine predictor into Scorer # You are currently rebasing branch 'preetam/outcome-predictor-rfc' on '91ddcce1'. # # Changes to be committed: # modified: doc/rfc/submitqueue/outcome-predictor.md # modified: doc/rfc/submitqueue/speculation-generator-best-first.md # modified: doc/rfc/submitqueue/speculation.md #
Document per-queue factor inheritance and the shared batch/path-set snapshot consumed by best-first ranking.
Fold ranking into one Score(ctx, batch, paths). Evidence is the scorer; heuristic/composite are the nested base. Frame the revision as a configured logit-linear model.
## Summary ### Why? The RFC dropped Predictor; the filename still said outcome-predictor.md. ### What? Move it to outcome-scorer.md and retarget links. Name the landing evidence factor after BatchStateLanding.
behinddwalls
force-pushed
the
preetam/outcome-predictor-rfc
branch
from
September 14, 2026 19:33
f98b8e8 to
2de5499
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
Ranking only needs one number per unresolved batch. A sibling Predictor factory duplicated the per-queue seam and made evidence look like a second public extension. The review asked for one ranking interface, framed as a logit-linear model.
What?
Scorer is the ranking extension:
Score(ctx, batch, paths). Evidence is the default impl wrapping a nested base (heuristic or composite). YAML istype: evidenceplusfactorsplusbase. The revision islogit(p') = logit(p_base) + Σ w_i x_iwith configured weights (w_i = log(factor_i)), not a fitted GLM.The RFC file is
doc/rfc/submitqueue/outcome-scorer.md. The evidence factor for the land-stage batch islanding, matchingBatchStateLanding.Test Plan
Docs-only change.
Stack