Repository navigation
Add Mem Off hackathon: claude-mem × cognee × HelixDB on a frozen coding benchmark - #62
Merged
Merged
Conversation
Add the Mem Off event folder: a head-to-head of memory systems for coding agents on a frozen eight-issue xitkit sequence (baseline 2fd55de, 11 failed / 769 passed on Python 3.14), with a bring-your-own-memory track and an extend-the-benchmark track. Why: memory plugins claim agents stop re-discovering the codebase, but nobody measures it on a task with a known answer. The protocol fixes the agent, model and repository, kills the agent session after every issue, and lets only the memory system carry context forward; the leaderboard is correctness-gated efficiency against a no-memory control. The harness builds on the cognee-report skill's `outcomes` protocol instead of a new evaluator: an agent adapter drives Claude Code headlessly, a sequencer chains the issues and emits evaluator-schema records so the skill's own `summarize` produces report.json. Submissions follow the earlier hackathons (one PR, one folder under submissions/) and a CI check refuses the mistakes seen in past PRs. The folder name carries a YYYY-MM-DD placeholder until the date is set; the named arms' manifests are drafts pending version pins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add a reporting skill for participants' coding agents, built on the cognee-hackathon-feedback pattern from PR #53: one installer drops the skill and a Stop-event checkpoint hook into the repo, appends an instruction block to AGENTS.md and CLAUDE.md, and starts a clock. Every ~15 minutes the agent updates memoff-report.md/.json from tool output only (harness run folders, the cognee-report scorecard, recall receipts, whether recalled items were used), and at wrap-up copies both files into submissions/<team>/report/. Why: the observation-skill output has to be submitted together with the action results, and participants should not have to remember to collect it. The CI check now requires the two report files for Track 1, the templates and docs describe them, and the cognee arm's receipt script no longer passes a flag cognee-evidence.py does not have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The harness tests run on every PR touching the event's harness, skills or submissions; the submission-folder rules apply only when a team folder under submissions/ changed, so organiser PRs are not rejected as malformed submissions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new event folder,
cognee-mem-off-hackathon-YYYY-MM-DD/, for Mem Off: a head-to-head of memory systems for coding agents (claude-mem, the cognee Claude Code plugin, HelixDB, and a bring-your-own-memory track) on a frozen eight-issue sequence against xitkit @2fd55de(baseline 11 failed / 769 passed on Python 3.14, verified). Event day 10:30–18:30. The folder name carries aYYYY-MM-DDplaceholder until the date is set.README.md— pitch, arms, the eight-issue table, run protocol, both tracks, PR submission flow, schedule, fairness ruleschallenge/— the eight Part A issues as separate files, the entrant contract, the adapter protocol, the metrics table, the known-gaps list for Track 2templates/—SUBMISSION.md, annotatedmemory.yml,ISSUE_PROPOSAL.md(seven sections)harness/— thememoffCLI: an agent adapter that drives Claude Code headlessly in thecognee-reportskill'soutcomesprotocol, and a sequencer that chains the eight issues per arm and repeat, ends the agent session between issues, runs the verifier (visible failure target + private grader), commits, waits for the memory system, and emits evaluator-schema records so the skill's ownsummarizebuildsreport.json. Also: single-issuepilotthrough the stock runner, the CI rules, manifest schema, draft arm manifests for cognee and claude-mem, a Dockerfile, 14 unit tests.skills/memoff-memory-report/— the reporting skill, built on thecognee-hackathon-feedbackpattern from feat: Add cognee-hackathon-feedback agent skill #53: one installer, aStop-event checkpoint hook, an instruction block inAGENTS.md/CLAUDE.md. Every ~15 min the agent updatesmemoff-report.md/.jsonfrom tool output only (harness run folders, thecognee-reportscorecard, recall receipts, whether recalled items were used) and copies them intosubmissions/<team>/report/at wrap-up. CI requires them for Track 1; they are submitted together with the Action'sresults/.submissions/README.md— one PR, one folder per team, what CI enforces, the exactresults/layoutexamples/— a smoke-runresults/folder, an example memory report, a worked issue proposal (all labelled synthetic / not evidence).github/workflows/memoff-submission-check.yml— harness tests +memoff check-submissionon submission PRsREADME.md— one table rowcognee-hackathon-feedback,memoff-memory-report) installed into the folderWhy
Memory plugins claim agents stop re-discovering the codebase; nobody measures it on a task with a known answer. The protocol fixes the agent, model and repository, kills the agent session after every issue, and lets only the memory system carry context forward. The leaderboard is correctness-gated efficiency against a no-memory control; issues 4, 5, 7 and 8 revisit code changed earlier and are the headline memory-delta metric. Building on the
cognee-reportskill rather than a new evaluator keeps scoring and reporting identical to what participants can run themselves.Submission rules follow the earlier events (folder-per-team PRs). About a third of past submission PRs edited the template in place, dropped a folder at the repo root, or committed
.env.example; the CI check refuses those.Verified
memoff run --smoke(two arms, two repeats, two issues) on the real checkout: real pytest counts per issue, records accepted by the skill'ssummarize,report.json,metrics.json, leaderboard, receipts, verify outputmemoff pilot --smokethrough the unmodifiedplugin_eval.py outcomes runmemoff check-submissiondry runs: accepts a well-formed Track 1 / Track 2 folder; rejects a template edit, a hand-writtenresults/,.env.example, and a missing memory reportNot in this PR (by design)
wipe/waitscripts not yet validated against the real plugins; the cognee plugin does not yet write injected context toMEMOFF_INJECTED_DIRcmemis a claude-mem fork,graphify-v8a Graphify fork)Before merge
RAW_BASEURL inskills/memoff-memory-report/install.shharness/memoff/data/prices.json/ the Dockerfile🤖 Generated with Claude Code