Skip to content

Add Mem Off hackathon: claude-mem × cognee × HelixDB on a frozen coding benchmark - #62

Merged
Vasilije1990 merged 3 commits into
mainfrom
mem-off-hackathon
Oct 17, 2026
Merged

Vasilije1990 merged 3 commits into
mainfrom
mem-off-hackathon

Conversation

@Vasilije1990

Copy link
Copy Markdown
Contributor

What

A new event folder, cognee-mem-off-hackathon-YYYY-MM-DD/, for Mem Off: a head-to-head of memory systems for coding agents (claude-mem, the cognee Claude Code plugin, HelixDB, and a bring-your-own-memory track) on a frozen eight-issue sequence against xitkit @ 2fd55de (baseline 11 failed / 769 passed on Python 3.14, verified). Event day 10:30–18:30. The folder name carries a YYYY-MM-DD placeholder until the date is set.

  • README.md — pitch, arms, the eight-issue table, run protocol, both tracks, PR submission flow, schedule, fairness rules
  • challenge/ — the eight Part A issues as separate files, the entrant contract, the adapter protocol, the metrics table, the known-gaps list for Track 2
  • templates/ — SUBMISSION.md, annotated memory.yml, ISSUE_PROPOSAL.md (seven sections)
  • harness/ — the memoff CLI: an agent adapter that drives Claude Code headlessly in the cognee-report skill's outcomes protocol, and a sequencer that chains the eight issues per arm and repeat, ends the agent session between issues, runs the verifier (visible failure target + private grader), commits, waits for the memory system, and emits evaluator-schema records so the skill's own summarize builds report.json. Also: single-issue pilot through the stock runner, the CI rules, manifest schema, draft arm manifests for cognee and claude-mem, a Dockerfile, 14 unit tests.
  • skills/memoff-memory-report/ — the reporting skill, built on the cognee-hackathon-feedback pattern from feat: Add cognee-hackathon-feedback agent skill #53: one installer, a Stop-event checkpoint hook, an instruction block in AGENTS.md/CLAUDE.md. Every ~15 min the agent updates memoff-report.md/.json from tool output only (harness run folders, the cognee-report scorecard, recall receipts, whether recalled items were used) and copies them into submissions/<team>/report/ at wrap-up. CI requires them for Track 1; they are submitted together with the Action's results/.
  • submissions/README.md — one PR, one folder per team, what CI enforces, the exact results/ layout
  • examples/ — a smoke-run results/ folder, an example memory report, a worked issue proposal (all labelled synthetic / not evidence)
  • .github/workflows/memoff-submission-check.yml — harness tests + memoff check-submission on submission PRs
  • root README.md — one table row
  • both agent skills (cognee-hackathon-feedback, memoff-memory-report) installed into the folder

Why

Memory plugins claim agents stop re-discovering the codebase; nobody measures it on a task with a known answer. The protocol fixes the agent, model and repository, kills the agent session after every issue, and lets only the memory system carry context forward. The leaderboard is correctness-gated efficiency against a no-memory control; issues 4, 5, 7 and 8 revisit code changed earlier and are the headline memory-delta metric. Building on the cognee-report skill rather than a new evaluator keeps scoring and reporting identical to what participants can run themselves.

Submission rules follow the earlier events (folder-per-team PRs). About a third of past submission PRs edited the template in place, dropped a folder at the repo root, or committed .env.example; the CI check refuses those.

Verified

  • xitkit baseline on Python 3.14: 11 failed / 769 passed
  • memoff run --smoke (two arms, two repeats, two issues) on the real checkout: real pytest counts per issue, records accepted by the skill's summarize, report.json, metrics.json, leaderboard, receipts, verify output
  • memoff pilot --smoke through the unmodified plugin_eval.py outcomes run
  • memoff check-submission dry runs: accepts a well-formed Track 1 / Track 2 folder; rejects a template edit, a hand-written results/, .env.example, and a missing memory report
  • 14 harness unit tests pass; hook time gate, loop guard and block decision covered

Not in this PR (by design)

  • Private graders and the reference implementation of the eight issues stay out of the public repo (drafts exist privately; untested until the reference branch is built — the critical path before the event)
  • Arm manifests for cognee and claude-mem are drafts: versions unpinned, wipe/wait scripts not yet validated against the real plugins; the cognee plugin does not yet write injected context to MEMOFF_INJECTED_DIR
  • The HelixDB arm is undefined until their team picks the memory layer (their cmem is a claude-mem fork, graphify-v8 a Graphify fork)
  • No run with a real model yet (needs the model pin and per-issue caps)

Before merge

  • set the date: rename the folder, fix the row in the root README and the RAW_BASE URL in skills/memoff-memory-report/install.sh
  • pin the model and Claude Code version in harness/memoff/data/prices.json / the Dockerfile

🤖 Generated with Claude Code

Vasilije1990 and others added 3 commits October 17, 2026 08:38
Add the Mem Off event folder: a head-to-head of memory systems for coding
agents on a frozen eight-issue xitkit sequence (baseline 2fd55de, 11 failed /
769 passed on Python 3.14), with a bring-your-own-memory track and an
extend-the-benchmark track.

Why: memory plugins claim agents stop re-discovering the codebase, but nobody
measures it on a task with a known answer. The protocol fixes the agent, model
and repository, kills the agent session after every issue, and lets only the
memory system carry context forward; the leaderboard is correctness-gated
efficiency against a no-memory control.

The harness builds on the cognee-report skill's `outcomes` protocol instead of
a new evaluator: an agent adapter drives Claude Code headlessly, a sequencer
chains the issues and emits evaluator-schema records so the skill's own
`summarize` produces report.json. Submissions follow the earlier hackathons
(one PR, one folder under submissions/) and a CI check refuses the mistakes
seen in past PRs. The folder name carries a YYYY-MM-DD placeholder until the
date is set; the named arms' manifests are drafts pending version pins.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add a reporting skill for participants' coding agents, built on the
cognee-hackathon-feedback pattern from PR #53: one installer drops the skill
and a Stop-event checkpoint hook into the repo, appends an instruction block
to AGENTS.md and CLAUDE.md, and starts a clock. Every ~15 minutes the agent
updates memoff-report.md/.json from tool output only (harness run folders,
the cognee-report scorecard, recall receipts, whether recalled items were
used), and at wrap-up copies both files into submissions/<team>/report/.

Why: the observation-skill output has to be submitted together with the
action results, and participants should not have to remember to collect it.
The CI check now requires the two report files for Track 1, the templates
and docs describe them, and the cognee arm's receipt script no longer passes
a flag cognee-evidence.py does not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The harness tests run on every PR touching the event's harness, skills or
submissions; the submission-folder rules apply only when a team folder under
submissions/ changed, so organiser PRs are not rejected as malformed
submissions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Vasilije1990
Vasilije1990 merged commit cf4f966 into main Oct 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant