How I stopped re-explaining the project to every AI agent

Every fresh AI session starts from nothing: it does not know my projects, the decision I made last week, or the patterns this codebase already follows, so I explain myself again, or I take a generic answer. The model is sharp and has no memory at all, at the same time.

I wrote about the personal side of this already: a self-hosted agent that carried my whole biography into every request, and how the fix was two shelves of memory, a small one always loaded and a big one searched on demand (I built an AI agent that knows me). This is the work side, where the problem was the opposite one, because the context existed, it just lived nowhere the agent could reach.

Decisions lived in chats

I joined a medical-travel product in January with very little written down. So I leaned on AI agents to work through the code with me, and they were good at it, but every session forgot everything: I would work something out, settle on a pattern, write it into CLAUDE.md, and next session I was explaining the same pattern again. I worked like that for about a month.

Then a docs folder in each repo

The first fix was the obvious one, a docs folder in each repository, mobile and backend, where I pasted plans and decisions from sessions. Within a month both folders were a mess, with the same feature documented in two places and old notes next to current ones, and I still had to tell the agent which file to open. It helped, but it was really me doing file search by hand and pasting the results back in.

One knowledge base, searched by meaning

So I pulled the docs out of both repos into one knowledge base shared into both, and that alone was only tidier. The change that mattered was the next step, indexing it for semantic search and giving Claude Code access to it over MCP, because now it retrieves by meaning: I ask, it pulls the relevant notes itself, and each one comes with a pointer back to where it came from.

Getting there was not clean, because gbrain, the open-source project I used for the index, is rough, and it took two days and three patches to fit my setup. The local embedding model crashed on the embedding step, so I patched that. A lot of my notes have Russian titles, and those were not indexed at all while Chinese, Japanese and Korean were handled fine, so I patched that locally too, and upstream has since fixed it properly, which means that patch is gone. And I keep my personal vault and the work knowledge base as two sources and wanted one search over both, which over MCP did not work, because it searched one default source unless the caller passed the right parameter, and the agent would not reliably pass it, so that was the third patch.

None of this is glamorous, but it is the actual work, because the model was never the blocker, the plumbing around the memory was.

The episode that convinced me

The app’s UI had drifted across AI sessions, not because anyone was careless, but because each session built what it was asked to build in isolation and did it slightly differently. The same input field shell got reinvented about nine times, cards had different radii, accents and shadows, and a single-select option looked one way on onboarding and another way on a hotel form.

The strange part is that the design tokens were in the code the whole time, there was a lint rule against hardcoded numbers and a line in CLAUDE.md saying the same, and the agent still reached for raw numbers instead of tokens.

What changed it was not a stricter rule but moving the design-system docs into the knowledge base, with a short orientation file written for agents rather than for people: find the semantic role of what you are building in the component catalog, use that component, and do not relitigate taste, because the taste calls are already made. Once I could tell the agent to pull that before writing any styles, new screens started matching the ones already there.

The design system itself got a proper rework along the way. I went through it with the ui-ux-pro-max skill, let it make the taste calls I am not a designer for, and locked the result, the tokens, the component catalog and the rules for using them, into the same knowledge base the agent reads first.

Smaller things followed the same pattern. Once I started fixing a bug and retrieval surfaced an old backlog note that was the real root cause, so we did the backlog task instead and the bug went with it. Business context works the same way: when the business side moves, I write it down and index it, so the agent knows why a feature matters, not just how to build it. I have even started recording my own QA walk-throughs out loud, transcribing them, and dropping the result into the same knowledge base as tasks.

A found note is a claim, not a fact

The knowledge base made the agent better, and it also made a new failure possible, because the agent now finds things, and a found note can be wrong about the code as it is today. I had a backlog note that said an address input did not exist on a form, and later it turned out the inputs did exist, as a dropdown, and the real cause of the problem was somewhere else, because the note had been written in good faith about an older tree.

So the rule I ended up with, for the agent and for myself, is that a retrieved note is a claim about the code and not the code itself, which means you check its date, check its status, and confirm it against the code by a different route than the note took, with different search terms and a different file. And if the search finds nothing, the agent should say so and not guess, because an answer that cites its source and admits a gap beats a fluent answer from a stale note.

Then specs on top of the knowledge base

The knowledge base fixed what the agent knows, but it did not fix on its own how a piece of work starts, because a good context layer with a vague task on top still produces a confident implementation of the wrong thing. So at some point I added OpenSpec on top of it: every change now starts as a written proposal with a design, the specs it touches and a task list, all generated by the agent from the knowledge base and read by me before any code exists, and the conventions I used to repeat by hand, the API rules, validation, i18n, testing, live in its config and get injected into every generation. The rule from the previous section is in there too, in writing: docs are claims, so re-verify them against the code by a different route.

The difference was bigger than anything I had changed before it. Mismatches with the design system or with the business requirements have simply not come up since, because they now get caught in the proposal, where fixing them costs a sentence instead of a rewrite.

Where I landed

The model still writes the code, and the knowledge base is what stops it from writing like it just walked in.

As of now the setup is two sources, the personal vault and the work knowledge base, federated behind one search, with OpenSpec on top of both for anything bigger than a small fix. I am still adding to it, but I am not going back to agents that start every session with amnesia. If your agents still start every session from zero, or you solved this in a different way, I would like to hear how it went, and LinkedIn is where I am easiest to reach.