Git Repos Are Not Enough Memory
The hard part is not getting an AI to answer. The hard part is getting the right context into the right place at the right time.
Not all context. Not a dump of notes. The relevant part. The current part. The part the agent needs for the task and is allowed to use.
That is why context engineering matters. Better prompts help, but better context systems matter more.
What I Use Today
My current memory system is simple: Markdown files in Git repos.
Project notes, rules, decisions, architecture docs, prompt snippets, source summaries, and AGENTS.md files live close to the work. Markdown is readable by humans and agents. Git gives version history, diffs, rollback, and a clear record of changes.
The most useful pattern is using separate repos as context boundaries.
In my setup, core holds governance and foundational context. Project repos hold working context. Agents get scoped access depending on the task. Some repos can be read-only. Others can be writable.
Why It Works
For one person, this is surprisingly strong.
It is simple, portable, and inspectable. It also creates a practical permission model: give the agent the few context areas it needs, not the whole system.
The setup is not fancy. That is part of the point. No database is required before a decision can be written down. No migration is needed before an agent can read a project note.
Where It Breaks
Git is not a knowledge management system.
It tracks file changes. It does not know which note is stale, which decision supersedes another, which fact only applies to one project, or which memory came from a validated source instead of a rough idea.
For code, this is manageable because there is an execution surface. The program builds, tests, and runs, or it does not.
Knowledge work is messier. A stale note can still look valid. A copied summary can lose its source. Three files can contradict each other and still pass every check. An agent can have access to thousands of files and still not know which ones matter right now.
This gets harder when multiple humans and agents share the same memory. The issue is not only retrieval. It is governance: who can read, who can write, what becomes durable memory, how conflicts are handled, and how the system explains why it retrieved a piece of context.
What I Want to Test Next
I still want to keep Markdown for a lot of use cases.
But I want to test a stronger layer underneath it:
- A database for structured facts, decisions, permissions, and timestamps.
- A vector database for semantic retrieval.
- A knowledge graph for entities, relationships, and superseded context.
Then wrap this in an API so agents and tools can ask for the right context package for a task.
The important part is not the stack itself. It is the operating model around it. A useful memory system should know where a memory came from, who can use it, when it was true, what replaced it, and whether two sources disagree.
Markdown files are a good start for personal AI memory.
Shared human-agent work needs something more active: context infrastructure that can retrieve, govern, update, and explain memory as the system changes.
That is the part I want to tinker with next.
Tinkering Queue
- Andrej Karpathy's LLM Knowledge Bases: a useful reference point for Markdown-based LLM knowledge bases.
- Graphiti: temporal knowledge graphs for agent memory.
- OpenAI Privacy Filter: useful for reducing sensitive data before it enters memory workflows.
- Privacy Filter model: the model behind the privacy filter.
- Pinecone: managed vector database for retrieval.
- Qdrant: open-source vector database to test locally or self-host.
- Ralph Gemini CLI extension: a small example of extending agent tooling around local workflows.