SDE-bench results for kgai, an AI memory benchmark for coding agents
Benchmark record. System under test: kgai 1.8.0, decision memory for AI coding agents. Benchmark: SDE-bench from the Agent Memory Benchmark (AMB). Result: 47, 48 and 51 of 61 bugs fixed on the first try over three runs, against 21 to 25 without memory. Raw data in the repository.
Summary
| 61 tasks, Claude Code with Claude Sonnet 5 | First-try fixes | Interventions per task | Cost per run |
|---|---|---|---|
| No memory (published by AMB, 5 runs) | 21-25 | 0.80-0.92 | $26-28 |
| Hindsight (published by AMB, 5 runs) | 40-46 | 0.28-0.41 | $19-22 |
| kgai 1.8.0 (3 runs) | 47, 48, 51 | 0.23-0.25 | $12.5-17.4 |
Across 183 task runs kgai solved 182 in the end and 146 on the first attempt. We build kgai, so this is a vendor's measurement. The benchmark, the harness and the comparison runs are not ours.
The benchmark
SDE-bench is the coding-agent benchmark in the Agent Memory Benchmark (AMB), an open AI memory benchmark harness. It has 61 bug-fix tasks in a Python codebase. Each task is built so that the obvious fix fails a hidden test, because the test encodes a decision the team made earlier and recorded only in a developer chat or a commit message. Examples: retry 5xx plus exactly 429 and 408 but no other 4xx, cap the attempts at 7 because that is what the rate-limit window measured, apply percent discounts before fixed ones.
The agent gets the repository and the bug report and writes a fix. The harness grades it against the failing test and the hidden test. A failed fix is reported back and the agent tries again, and each of those rounds counts as an intervention. The headline metric is the number of tasks fixed with zero interventions.
Each task carries a corpus of past project knowledge: the chat or commit that holds the decision, 140 decoy developer chats and 100 real commits from the host repository as noise. This is what separates it from conversational AI memory benchmarks. It does not ask whether a fact can be retrieved, it asks whether the agent gets the change right.
System under test
kgai is decision memory for AI coding agents, an AI decision storage layer. An agent records each engineering decision with its reason into an immutable log, and later agents read the decisions currently in force before they change code. Recall is lexical plus graph traversal with no embeddings. Version 1.8.0, open source under MIT.
Method
Agent and model: Claude Code with Claude Sonnet 5, the harness defaults, identical in every arm.
Capture. kgai stores decisions, not documents, so the corpus was first recorded the way it is in normal use. The same model, with kgai's stock knowledge-graph skill and no benchmark-specific instructions, read every corpus document as a past session and recorded whatever that session decided. It never saw the hidden tests or the task descriptions. 240 shared documents produced 21 decisions, with most decoys skipped. 66 task documents produced 84 decisions. For 6 tasks nothing was recorded. One-time cost: $33 for all 306 documents, reused by all three runs.
Retrieval. Before each task the harness passed the agent the output of kg context --about "<bug report>" through its generic memory arm, the slot any memory system can use. A typical answer held one to six elements, each with its current decision and reason, a few hundred words in all.
Runs. Three full runs of all 61 tasks.
Results by run
| Run | First-try fixes | Solved in the end | Interventions per task | Cost |
|---|---|---|---|---|
| 1 | 47 of 61 | 61 | 0.25 | $12.51 |
| 2 | 48 of 61 | 61 | 0.25 | $16.56 |
| 3 | 51 of 61 | 60 | 0.23 | $17.38 |
Cost is the API-equivalent figure Claude Code reports per run. It excludes the one-time capture. As far as their result files show, the published comparison runs count the agent only as well. Ten tasks were not fixed on the first try in any run. One of them is a MIME-type table with half a dozen exceptions.
Limits
Three runs, and results vary between runs. The published Hindsight runs alone span 40 to 46. One codebase in one language. The capture step used the same model as the agent. Decisions that existed only as a commit titled "fix: …" were not recorded, because the skill deliberately skips plain bug fixes, and that cost a few tasks. The result is self-reported. AMB lists only results its maintainers reproduce, and SDE-bench is not on its public leaderboard.
Data and reproduction
Scripts, the harness patch for Linux, per-task results and the exact memory each agent received are in bench/sde-bench. Comparison runs are in the AMB repository under outputs/sdebench. Related: the million-decision scale test.
Try it on your repo. Two commands, MIT, nothing leaves your machine until you configure a bucket.
See how kgai compares to other tools on the comparison page, or browse the other notes.