Garry Tan Ships GBrain Evals — But Who Grades the Grader?

Garry Tan, President and CEO of Y Combinator, spent 2026 publicly building gstack (his Claude Code skill pack) and GBrain (his long-term memory layer for OpenClaw). On August 31, 2026, he shipped a new artifact for GBrain: a public eval suite meant to prove the memory system reads its own stored data back accurately, and writes new memories well from raw conversation. The claim is state-of-the-art. The most-liked reply asks the one question that actually matters before anyone should believe it.

What Tan posted

From @garrytan on X, verbatim:

"I just made some new GBrain evals that help prove that my retrieval-for-AI-agent open source layer is SOTA for reading memory back without LLM-in-loop

And I've also added evals for memory-save from agent transcript, which is the best way to enrich your brain

Receipts at https://github.com/garrytan/gbrain-evals"

TL;DR

GBrain, in one paragraph, and how it relates to gstack

GBrain is not new — Tan described building it back in May, after shipping gstack and Gary's List, his AI-powered blogging platform. The origin story, in his own words from that period: "I could just open this project in conductor and then the first thing I write is like, go look at Gary's List, look at how we do chunking, embedding, hybrid RRF, RAG—extract it and use it for my OpenClaw." The result is a RAG system purpose-built as the long-term memory an AI agent reads from and writes to — vector embedding plus Hybrid RRF (Reciprocal Rank Fusion) reranking, backed by PostgreSQL with pgvector.

GBrain and gstack are siblings, not the same project. gstack is the slash-command sprint process (/office-hours, /review, /ship); GBrain is the memory substrate an agent consults across sessions. The one documented link between them: gstack's ./setup --host installer lists GBrain among the environments its skills can be installed into, alongside Codex, Cursor, and OpenClaw itself. That makes GBrain a peer host in Tan's ecosystem, not a gstack component — worth getting right, since the two names get conflated in casual coverage.

What the new evals actually measure

Per the repository, gbrain-evals breaks into three benchmark categories:

1. Retrieval performance (read path). GBrain reports 97.6% R@5 (recall at 5 results) on the public LongMemEval dataset — the same benchmark family explainx.ai has covered before in a different agent-memory project — achieved without an LLM in the retrieval loop. Concretely, that means the system ranks and returns candidate memories using embeddings and keyword matching alone, with no model call to rerank results before returning them. That's the "reading memory back without LLM-in-loop" claim from the launch tweet: it's a cost and latency argument as much as an accuracy one — a memory lookup that doesn't burn an LLM call on every agent turn is cheaper and faster if its recall holds up.

2. Write-path quality ("Cat 35"). A separate eval measures whether salient content from a raw agent working session survives into GBrain's stored memory pages: the repo reports 88.1% of salient content preserved, 91% of resulting pages usable, and zero junk leakage. This is the "memory-save from agent transcript" Tan's second tweet describes — the mechanism of extracting durable facts and preferences out of a messy conversation log rather than storing the transcript verbatim.

3. Proactive memory volunteering ("Cat 34"). A third eval checks whether GBrain surfaces relevant memories unprompted at the right moment in a session, reporting "0 know-to-ask failures" and "push precision 1.0."

The repo's methodology description leans on genuinely good practice on paper: sealed answer keys the system under test can't access during scoring, pinned commits, public datasets, and published errata for discovered issues. None of that is nothing. It's also not the part of the story that's actually in question.

The question that matters: who wrote the test?

The single most useful reply in the entire thread came from Elev (@Elev_30):

"the eval set and the retrieval layer have the same author. what does gbrain score on a memory benchmark somebody else wrote?"

That's not a snarky dunk — it's the correct methodological objection, stated as plainly as it can be stated. GBrain and gbrain-evals are both Garry Tan's own repositories. The same team that built the retrieval layer also built the sealed answer keys, chose the corpora, and decided what counts as a correct retrieval or a "salient" fact worth preserving. Sealed keys and pinned commits are good practice for reproducibility — anyone can rerun the eval and get the same number — but they say nothing about independence. A test can be perfectly reproducible and still be built by someone with every incentive to write it in a way their own system passes.

This is not a hypothetical concern specific to GBrain. explainx.ai flagged the identical pattern the same day, in a completely unrelated launch: Abliteration.ai's hosted GLM-5.3 shipped cyber-capability numbers that traced back entirely to the model vendor's own self-reported testing, with no outside party having verified them. Two unrelated products, two unrelated teams, the same structural gap: a benchmark built by the entity being benchmarked is a claim, not a verification — regardless of how legitimate the underlying tool is. It's worth naming as a pattern because it's becoming a recognizable one in AI tooling launches broadly: ship the product, ship your own eval suite for the product, present the second as proof of the first. The fix isn't "distrust every vendor number" — it's "wait for, or go run, an independent reproduction before repeating the number as fact."

One data point worth naming directly: GBrain's 97.6% R@5 on LongMemEval sits close to 96.6% R@5, the number reported by MemPalace, a different agent-memory project explainx.ai covered in April 2026 — which went viral on the same claim ("highest-scoring AI memory system ever benchmarked") and then had to walk back several overstated numbers after a GitHub issue thread picked the README apart. That's not evidence GBrain's number is wrong. It is a reminder that "we scored well on LongMemEval" is a claim this exact corner of the industry has already had to publicly correct once this year, on a nearly identical margin.

The framing worth building on: curation is the hard part, not search

Buried in the replies is a genuinely useful reframe from Teneo Protocol:

"Most people think agent memory = vector search. It isn't. The real challenge is deciding what deserves to survive and retrieving it at exactly the right moment."

This is worth taking seriously on its own, independent of any GBrain-specific question. Vector search — embed a query, find nearest neighbors — is a solved, commoditized problem; every framework has a version of it. The two problems that remain genuinely hard are:

• Curation: out of everything that happened in a session, what actually deserves to become durable memory? Store too much and you drown future retrieval in noise (or leak sensitive raw transcripts, the "store everything, score high" trap explainx.ai flagged in the MemPalace piece). Store too little and the agent forgets things it needed.
• Timing: even with the right fact stored, retrieving it at the right moment — not just when a keyword happens to match, but when it's actually relevant to what the agent is doing — is a different, harder problem than raw recall@k on a static test set.

GBrain's "Cat 35" write-path eval and its "Cat 34" proactive-surfacing eval are direct attempts at exactly this — they're evaluating curation and timing, not just search. That's the right instinct. Whether 88.1% preserved and "push precision 1.0" hold up outside GBrain's own sealed test set is the open question the rest of this post is about.

The staleness problem nobody's evals cover yet

A second sharp, substantive reply, from No Body:

"short long term mem? who validates the memory? what if faulty mem is stored but never updated?"

This is a real gap, and it's not unique to GBrain — it's an open problem across agent-memory tooling generally. Every long-term memory system has a write path (save this fact) and a read path (retrieve relevant facts). Almost none of them, GBrain included as far as the published evals show, have a systematic re-validation path: a mechanism that periodically checks whether a stored fact is still true, flags contradictions when new information conflicts with old memory, or expires facts that have a natural shelf life.

Concretely: if an agent stores "the user's staging server is at IP X" in March and the server moves in July, nothing in a standard retrieval-precision benchmark catches that the stored fact is now wrong — R@5 measures whether the system found the relevant memory, not whether the memory is still true. This is a distinct failure mode from a retrieval miss, and it needs a distinct eval category (something like a "memory contradiction / staleness" benchmark) that, as of this launch, doesn't appear to exist for GBrain or for most competing memory layers.

Multi-user memory: the open question two people asked and nobody answered

Two separate replies raised the same underlying gap from different angles:

• threadkillerokc, describing an active company deployment: "I'm in the process of helping my firm deploy gbrain company-wide... We are looking deeply into: 1 - multi-user access to a single brain (hundreds of people, multiple source ownership, etc.)..." (the reply cuts off in the source thread, but the direction is clear — this is a practitioner already trying to run GBrain at organizational scale and hitting the multi-tenant question head-on).
• Dee, more bluntly: "did you implement 'team brain' yet?"

Neither got an answered reply in the thread. That matters because single-user memory and multi-user memory are not the same engineering problem scaled up — they're different problems. A single-user brain has one source of truth and one access-control question (is this the user's own agent). A "team brain" with hundreds of people and multiple source-of-truth owners needs answers to questions the current eval suite doesn't touch: who can write to shared memory, whose facts win when two people's stored preferences conflict, and what happens to team memory when someone leaves. explainx.ai covered a competing product tackling exactly this shape of problem — TencentDB's Agent Memory v2 ships a "team hub" model for shared chat, skills, wiki, and code-graph memory — worth reading as a comparison point for anyone evaluating GBrain against what a multi-user memory layer needs to actually cover.

Honest limitations

• No independent reproduction of any GBrain-evals number as of publication. Treat 97.6% R@5, 88.1% preserved, and "push precision 1.0" as vendor-reported claims, the same caveat explainx.ai applied to Abliteration.ai's cyber benchmarks the same day.
• No staleness/re-validation eval published. The suite tests read accuracy and write quality at a point in time, not whether stored memory stays correct as facts change.
• No multi-user/team-brain eval or feature confirmed. Two thread replies asked about it directly; neither got a public answer.
• LongMemEval proximity to a previously-corrected competitor's number. 97.6% sits close to MemPalace's 96.6%, a figure whose surrounding marketing claims required a public walkback in April 2026 — not proof GBrain's number is wrong, but a reason to want independent verification before treating it as settled.

Related on explainx.ai

• gstack: Garry Tan's open-source "software factory" for Claude Code — the sibling project and skill pack GBrain lists as a supported host
• How Gary Tan Shipped 400x More Code Using Claude Code & AI Agents — GBrain's origin story, in Tan's own words, including the Karpathy "knowledge LLM wiki" inspiration
• Abliteration.ai Hosts an Uncensored GLM-5.3 for Offensive Cyber Work — the same self-graded-benchmark pattern, published the same day, in a completely unrelated product
• MemPalace, LongMemEval, and what Reddit got right about the viral "highest-scoring" AI memory repo — the closest LongMemEval comparison point, and a cautionary tale about memory benchmarks that don't survive scrutiny
• Karpathy LLM Wiki: The Pattern Behind Agent Memory — the conceptual pattern GBrain was built to implement
• TencentDB Agent Memory v2: Team Hub for Chat, Skills, Wiki, CodeGraph — a competing product's answer to the "team brain" question GBrain hasn't answered yet
• RAG vs MCP: The Complete Guide to Context-Aware AI Systems — background on the retrieval architecture GBrain builds on

Resources

• Evals repo: github.com/garrytan/gbrain-evals
• gstack repo: github.com/garrytan/gstack

Accuracy note: This post reflects Garry Tan's launch thread on X (published August 31, 2026) and the public gbrain-evals repository, current as of publication on September 1, 2026. All benchmark figures cited are Tan's own team's self-reported numbers unless otherwise noted; none have independent, third-party reproduction confirmed as of publication.