I built a second brain for my AI agents. The hard part wasn't the AI.

Engineering · Sep 2026 · by Ascendra Labs

I spent a month teaching my AI tools to remember who I am and how my world works. The interesting part turned out to be the boundary: where to let the model decide, and where to never let it.

This started with an uncomfortable audit. I exported about three months of my own chats with AI coding tools and counted what I was actually doing in them. Roughly seven thousand prompts. A large share of that was me re-typing the same context over and over: how my environment is set up, which database tool I use, how a particular deploy works. I had explained the same handful of things hundreds of times. And every good decision I made in those sessions, the ones with real reasoning behind them, vanished the moment I closed the window.

So I built a memory the tools could read on their own. It loads the right context when a session starts, and it writes decisions back when a session ends. Underneath, it is two layers that do not really trust each other: a deterministic layer of plain files and git, and a non-deterministic layer of model judgment sitting on top. Almost every hard problem I hit came down to one question. Where exactly does the line between those two layers go? These are the answers that surprised me.

The model goes in the middle, never on the edges

One of my first rules was blunt: no language model inside the tools themselves. Tools are plain code you can test and trust. The model can help me decide what to do, but it does not get to be the thing that runs.

Then I hit a job that plain code is genuinely bad at: reading the layout of a document. I had hand-written a parser for one kind of statement, and it broke the second a different format showed up. A stricter version was worse. It read every field one column off and reported numbers that were confidently, uniformly wrong. Nothing crashed. It just lied.

The fix was to let the model do the one thing it is good at, and box it in on both sides. Plain code decides whether a document even qualifies. The model reads the messy layout and hands back structured numbers. Then plain code checks those numbers against an accounting identity that has to balance, and throws the whole thing out if it does not. The model never gets the last word on a figure. It sits in the middle, with a deterministic gate in front of it and another one behind.

A confident wrong answer is worse than a blank

That parser bug taught me something I now treat as a law: a plausible wrong answer is a more dangerous bug than a missing one. A blank is honest. You see the gap and you go fill it. A wrong number that looks right sails straight past you and quietly poisons everything downstream.

So I gave every extractor an explicit way to give up. If it cannot find a value with confidence, it returns nothing and flags the item instead of guessing. Where two readings are possible, it picks one by a fixed rule rather than a coin flip. A running balance, for instance, always takes the last figure on the page, never the first. And when it genuinely needs a value it does not have, it looks up how similar things were handled before rather than inventing one. Refusing to answer is a feature, and it took me longer than I would like to admit to build it on purpose.

Fail open or fail closed is a decision, made one control at a time

A few pieces of this system sit in the path of every single action. The bit that loads my context at the start of a session. The guard that watches for secrets leaving. My instinct was to pick one philosophy and apply it everywhere. That instinct was wrong.

The context loader fails open. If it breaks, the session just starts with less memory, and I would much rather that than a tool refusing to run because a helper hiccuped. The secret guard fails closed. If it cannot tell whether something is safe, it blocks. I learned the difference the hard way, when an early version of that guard managed to brick every command in a session two different ways at once, once by being too strict and once by crashing on its own error. The rule that came out of it stuck: for anything convenient, a bug should cost you a feature; for anything protective, a bug should cost you the action. Do not globalise one policy.

You cannot leak what you have no tool to send

I wanted one of these agents to be reachable from my phone, as a running conversation that could actually look things up in my own data. The obvious worry is prompt injection: some text it reads tells it to quietly ship my information somewhere. You can try to instruct a model not to do that. You will lose that argument eventually.

The move that actually worked was to take the tools away. This agent is read-only by construction, and it has no way to reach the open internet. No fetch, no send. It literally does not possess a tool that could carry data out. You can inject whatever you like into what it reads, and there is still no exit. It turns out the strongest instruction you can give a model is an empty toolbox. You just do not hand it the capability in the first place.

On one computer, separate folders protect nothing

Early on I kept my knowledge in one place and my secrets in another, and felt good about the separation. Then I noticed the obvious. Every one of these tools runs as me, on my machine, with my permissions. A folder boundary between two things that share one user is a fence with no gate. Anything that can read one can read the other.

Real safety had to come from the shape of the data, not from where it sits. So the brain holds references to secrets, never the secrets themselves. Every entry is marked as something the agent must never read, and points at where the value actually lives instead of containing it. When a tool genuinely needs the value, it is resolved inside a wrapper the agent cannot see into, the same way you never hand your SSH key to anything and simply let SSH use it for you. The value never lands anywhere the model can see it.

"Deterministic" is not free

I kept calling the knowledge graph deterministic. Same inputs, same output, rebuild it any time you like. Then I noticed the clusters inside it kept shifting between rebuilds even when nothing had changed. The cause was almost funny. A step walked over a set of text keys, and the language runtime randomises that order on every run for security reasons. The thing I had been calling reproducible was quietly reshuffling itself.

The fix was one line, pinning the seed that controls the ordering. The lesson was bigger than the fix. Deterministic is a claim you have to actually earn. Hidden ordering, wall-clock time, anything random by default, they will all smuggle a little chaos into something you swore was stable. If you depend on a rebuild coming out the same, go and check that it really does.

You can get database guarantees out of plain files

The whole store is just files and git. No database server anywhere. I love that for how simple and portable it is, but it meant I had to earn the guarantees a database would have handed me for free.

Two writers saving a record at the same instant could compute the same next number and quietly overwrite each other. The fix is an old trick: create the file in a mode that fails if it already exists, and on a clash, recompute and try again. A silent overwrite turns into a loud retry. I also stopped keying records on a project's short name, because two different projects can share one, and switched to a key built from the git remote so it survives renames and moves. And the sharpest lesson was about the test for that overwrite bug. It passed against the broken code, because the bug needed two writers to collide at exactly the wrong moment. A race test is worthless until you have watched it fail on purpose. I had to force the collision by hand before I could trust the fix.

Three smaller ones I keep coming back to

The knowledge graph is disposable on purpose. The real source of truth is a pile of hand-written text files. The graph is just a cache I can rebuild from them in one command, so I never back it up and I am never scared to delete it. When the database engine I had picked got quietly abandoned by the people who made it, swapping it out was a non-event, because nothing important lived there in the first place.

Sometimes repeating yourself is the right call. I have two small functions that filter transactions and look almost identical. Every instinct says merge them. They must never be merged. One counts money moving between my own accounts, the other has to ignore exactly that, and their correctness is opposite. I left a written warning next to them so that a tidy-minded future version of me does not clean it up and break both at once.

Moving a leaked secret does not un-leak it. At one point the plan was to move some old keys into a proper vault and call it fixed. But if a key already sat in git history, relocating it forward changes nothing about the fact that it is already out there. Tidier storage from now on and cleaning up a past exposure are two different jobs. For a secret that has already leaked, the only real fix is to rotate it.

If there is a thread through all of this, it is that the interesting work was not the AI. It was deciding, over and over, where to trust it and where to build a wall. The model is fast and fluent and occasionally, confidently wrong, so the craft is in bracketing it with plain, boring code that cannot be talked out of the truth.

And the only reason I could write any of this down is that the brain was writing itself down the whole time. Every session ended with a one-line note of what I decided and why. I did not have to remember. That was rather the point.