[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21
Open
crystalxyz wants to merge 4 commits intoharbor-framework:mainfrom
Open
[Ready for Review] Adapter | Condense tutorial for human, create agent version separately#21crystalxyz wants to merge 4 commits intoharbor-framework:mainfrom
crystalxyz wants to merge 4 commits intoharbor-framework:mainfrom
Conversation
crystalxyz
changed the title
Ternura143
suggested changes
Mar 25, 2026
Ternura143
left a comment
Ternura143
left a comment
There was a problem hiding this comment.
LGTM! I only have a small issue that needs to be addressed.
| | `split` | string | yes | Split name matching original. Use `"full"` if adapter works for all splits collectively. If different splits are registered/validated in different ways, split them out separately. | | ||
| | `adapted_benchmark_size` | int | yes | Number of tasks the adapter can convert. May differ from original if tasks were excluded for sufficient reasons documented in the README. | | ||
| | `parity_benchmark_size` | int | yes | Number of tasks used for parity. Equals `adapted_benchmark_size` if full set. | | ||
| | `parity_sampling_rate` | float | yes | `adapted_benchmark_size / parity_benchmark_size` | |
There was a problem hiding this comment.
parity_sampling_rate formula is inverted in the schema — should be parity_benchmark_size / adapted_benchmark_size, not adapted_benchmark_size / parity_benchmark_size.
AlienKevin
reviewed
Mar 26, 2026
|
|
||
| ### Step 3: Verify Oracle Solutions | ||
|
|
||
| Run the oracle agent on your entire dataset and confirm **100% reward on all tasks**. |
There was a problem hiding this comment.
For cases where the original benchmark has broken oracles, maybe we can advice the agent to document the tasks with oracle issues and file bugs to the upstream benchmark instead of attempting to fix them on the Harbor side.
crystalxyz
force-pushed
the
adapter-tutorial-fix
branch
from
6a77ccd to
99eb1f1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR was motivated by a prior Harbor meeting discussion saying that the current harbor adapter tutorial is really long and hard to follow for human readers. In order to lower the entrance boundary and make it easier for people to contribute, we decide to condense the adapter tutorial for human readers so that they can easily keep track of their progress. Also, a separate agent-versioned tutorial is created so that contributors can easily use agents to build adapters.
NOTE: New structure updates (Mar 28, 2026)
/doc/datasets/adapter-human(Title:Adapters (Human Guide)) -> Human version tutorial/doc/datasets/adapter(Title:Adapters (Agent Guide)) -> Agent version tutorialDesign details
/adapterurl is reserved for agents with the most comprehensive information, with a callout box to redirect human readers to a more concise website. In this way, all the adapter readers will be aware of the agent website so that they can use it to build their adapter.adapters.mdxfile is used to host agent guide, because it makes git history and changelogs easier to followadapters-human.mdxfile is used to host human guide, and this is created from scratch to show a concised version of tutorial steps