# SDE-bench results for kgai, an AI memory benchmark for coding agents

*SDE-bench AI memory benchmark. Claude Code with kgai decision memory fixed 47 to 51 of 61 bugs on the first try, against 21 to 25 without memory.*

*Benchmark record. System under test: kgai 1.8.0, decision memory for AI coding agents. Benchmark: SDE-bench from the Agent Memory Benchmark (AMB). Result: 47, 48 and 51 of 61 bugs fixed on the first try over three runs, against 21 to 25 without memory. Raw data in the repository.*

## Summary

| 61 tasks, Claude Code with Claude Sonnet 5 | First-try fixes | Interventions per task | Cost per run |
|---|---|---|---|
| No memory (published by AMB, 5 runs) | 21-25 | 0.80-0.92 | $26-28 |
| Hindsight (published by AMB, 5 runs) | 40-46 | 0.28-0.41 | $19-22 |
| kgai 1.8.0 (3 runs) | 47, 48, 51 | 0.23-0.25 | $12.5-17.4 |

Across 183 task runs kgai solved 182 in the end and 146 on the first attempt. We build kgai, so this is a vendor's measurement. The benchmark, the harness and the comparison runs are not ours.

## The benchmark

SDE-bench is the coding-agent benchmark in the Agent Memory Benchmark (AMB), an open AI memory benchmark harness. It has 61 bug-fix tasks in a Python codebase. Each task is built so that the obvious fix fails a hidden test, because the test encodes a decision the team made earlier and recorded only in a developer chat or a commit message. Examples: retry 5xx plus exactly 429 and 408 but no other 4xx, cap the attempts at 7 because that is what the rate-limit window measured, apply percent discounts before fixed ones.

The agent gets the repository and the bug report and writes a fix. The harness grades it against the failing test and the hidden test. A failed fix is reported back and the agent tries again, and each of those rounds counts as an intervention. The headline metric is the number of tasks fixed with zero interventions.

Each task carries a corpus of past project knowledge: the chat or commit that holds the decision, 140 decoy developer chats and 100 real commits from the host repository as noise. This is what separates it from conversational AI memory benchmarks. It does not ask whether a fact can be retrieved, it asks whether the agent gets the change right.

## System under test

kgai is decision memory for AI coding agents, an AI decision storage layer. An agent records each engineering decision with its reason into an immutable log, and later agents read the decisions currently in force before they change code. Recall is lexical plus graph traversal with no embeddings. Version 1.8.0, open source under MIT.

## Method

Agent and model: Claude Code with Claude Sonnet 5, the harness defaults, identical in every arm.

Capture. kgai stores decisions, not documents, so the corpus was first recorded the way it is in normal use. The same model, with kgai's stock knowledge-graph skill and no benchmark-specific instructions, read every corpus document as a past session and recorded whatever that session decided. It never saw the hidden tests or the task descriptions. 240 shared documents produced 21 decisions, with most decoys skipped. 66 task documents produced 84 decisions. For 6 tasks nothing was recorded. One-time cost: $33 for all 306 documents, reused by all three runs.

Retrieval. Before each task the harness passed the agent the output of `kg context --about "<bug report>"` through its generic memory arm, the slot any memory system can use. A typical answer held one to six elements, each with its current decision and reason, a few hundred words in all.

Runs. Three full runs of all 61 tasks.

## Results by run

| Run | First-try fixes | Solved in the end | Interventions per task | Cost |
|---|---|---|---|---|
| 1 | 47 of 61 | 61 | 0.25 | $12.51 |
| 2 | 48 of 61 | 61 | 0.25 | $16.56 |
| 3 | 51 of 61 | 60 | 0.23 | $17.38 |

Cost is the API-equivalent figure Claude Code reports per run. It excludes the one-time capture. As far as their result files show, the published comparison runs count the agent only as well. Ten tasks were not fixed on the first try in any run. One of them is a MIME-type table with half a dozen exceptions.

## Limits

Three runs, and results vary between runs. The published Hindsight runs alone span 40 to 46. One codebase in one language. The capture step used the same model as the agent. Decisions that existed only as a commit titled "fix: …" were not recorded, because the skill deliberately skips plain bug fixes, and that cost a few tasks. The result is self-reported. AMB lists only results its maintainers reproduce, and SDE-bench is not on its public leaderboard.

## Data and reproduction

Scripts, the harness patch for Linux, per-task results and the exact memory each agent received are in [bench/sde-bench](https://github.com/kgaidev/kgai/tree/main/bench/sde-bench). Comparison runs are in the AMB repository under outputs/sdebench. Related: [the million-decision scale test](https://kgai.dev/blog/million-decisions-benchmark).

## Try kgai

Two commands, MIT, nothing leaves your machine until you configure a bucket:

```
claude plugin marketplace add kgaidev/kgai
claude plugin install kgai@kgai-marketplace
```

Source: https://github.com/kgaidev/kgai
See how kgai compares to other tools: https://kgai.dev/compare
All notes: https://kgai.dev/blog/

---

Canonical: https://kgai.dev/blog/sde-bench-ai-memory-benchmark · Written by kgai maintainers · kgai is MIT open source