Home

LLM VAULT GARDENERS

My vault has slowly morphed into a workflow that I am happy with. It feels sustainable. I tried PARA, MOCs, Zettelkasten, with varying degrees of success. I always had the nagging feeling that as my vault grew to over 1000 notes that some of them would never be seen again. This is my attempt to ameliorate that. It is based on the most excellent Karpathy system. Consider it a fork?

I do not have as much time as I like to interact with my vault, work and family take precedence so I needed a way to track the flow of my vault. My new workflow now captures my chats, links every new note into the correct place in the vault using a combination of embedding and Graphify, and maintains topic hubs that grow and die on their own. All of it runs hourly on a cheap server. I no longer curate my hubs. My topic hubs are there for me whenever I can find the time.

HOW I GOT HERE

It started with Obsidian Copilot and having a llm to chat with in my vault. I was building up substantial chats history with Copilot, some good ideas, some llm bullshit, typical llm interactions. When Copilot 4 with Copilot Agents arrived I decided to mine my old chats. This took some time as I was checking the output each time, but it was a good process and gathered much thinking that I would have lost.

It led to new build step 1:

STEP 1: LIVE CHAT MINING

I built instructions into the Copilot Agents - as I chat it creates a note. This ended up in my Inbox where my clippings and external material arrive. It seemed like the right place and turned out to be the right place to fit with the rest of the build. That was more by happy coincidence than any advance planning on my part!

The Chat Miner captures my responses in the chat verbatim, and then summarises the llms points. The llm is in callout so I can easily see distinguish between human and llm. The Inbox acts as a staging area. Each captured note has a tickbox, the note does not get processed until I review, edit, and approve by ticking the note. Approval moves the note to step 2 processing where it is processed and archived.

One thing I had to add retrospectively is that the note gets linked to the chat it came from. If I ever pick up a chat from my history the note comes out of the archive so there is a built in de-dup to avoid noise.

STEP 2: INBOX PROCESSING

When I set this up my inbox only contained web clippings. Luckily the workflow fitted perfectly with the Chat Miner notes! I thought it was the sign of a good system that it could handle information no matter the provenance.

This is the llm summary of the processing steps because it is far better at summarising skills that I am!

LLM

  • Scope. The run lists the inbox and classifies every item: web clip, converted PDFs, unconverted PDF, or Chat Mining. Unconverted PDFs are flag-only — never processed, moved, or parsed. A run caps at 12 items; the rest wait for the next hour. The run re-reads the inbox immediately before any write, because Martin edits the vault in parallel and its earlier observations can be stale. Private material, publishing folders, and tool-output zones are never in scope.
  • Triage. Items are split into chunks of ~6, one read-only subagent per chunk, all dispatched in parallel. Each agent returns what the item is, a 2–3 sentence substance summary, near-duplicate clusters with the canonical item named, topic candidates, and whether the item is idea-rich enough to spawn atomic notes. Triage reports are input; the main agent spot-checks every marginal and no-fit call by reading the item itself.
  • Fit. Three verdicts per item, decided by the run with no confirmation round-trip. Corpus fit maps to a known topic or thread and gets full processing. Marginal means interesting with no settled home: a minimal reference note, flagged in the run summary. No fit or duplicate goes straight to a dated trash folder (logged, reversible). A dup cluster’s canonical item is processed; its alternates are trashed. Renames are applied automatically.
  • Surgery. Converted papers and image-bearing clips get a deterministic asset pass: backup first, image links rewritten to bare-filename wiki links, URL-encoded characters decoded, images moved to a central images folder after a name collision check. Remote URLs are never touched. Dangling image references are report-only — never deleted or “fixed”.
  • Process. Corpus-fit items become notes in the permanent corpus with full frontmatter and a source link, so the original can always be re-pulled online. Raw scraped titles are renamed to a standardised “type — title — date” format. Idea-rich items spawn claim-titled atomic notes, cross-linked to each other, each linking back to its origin. Marginals get a fit note stating why they are marginal and what would promote them.
  • Link. The step that turns placed notes into a connected corpus. Candidates come embedding-first from a vector index the server rebuilds hourly, get verified on disk, then land as 2–5 strong links plus reciprocal inline patches into genuinely relevant existing notes. My prose is never edited beyond link lines.
  • Lint + review. The run is complete only when every finding is disposed. Programmatic lint over every touched file: frontmatter parses, links resolve, the log entry is intact, claimed files exist. A fresh-context reviewer audits judgment: claim-to-source coverage on atomic notes, patches staying link-lines, the angle meeting its bars. Mechanical findings are fixed and re-linted; judgment findings are fixed and re-reviewed; anything genuinely mine to decide comes back in the closing summary.
  • Log. One entry in an operations log per run, newest at top: fit tally, renames, angle proposals, reviewer outcome. The run verifies the inbox is empty, or says exactly what remains and why. A pattern scan offers (never applies) a standalone note when a recurring cross-clip pattern appears.

I have my vault synced, headless Obsidian sync, to a cheap server from Hetzner as part of an idea that ended up failing. I hooked up my vault to Open WebUI so I could interact with my vault from any machine. OWUI lacked the controls of Copilot agents and messed up my whole vault leading to having to recover my vault from a previous synced version - lesson learned, stick with Copilot. It did mean that I had a server with opencode (glm 5.3-flash running serverless on TensorX, an Irish ZDR, GDPR, inference provider; thumbs up, very happy with them!).

The always on server gave me an idea that I could run cron jobs on my vault to process my inbox which at that time contained only clippings from websites, but now also has the chat mining notes. The machine is always on, syncs across my other machines, and seemed like the obvious place to run this from. Learning my lesson from OWUI I set up a git tracker for the vault so any server issues were recoverable from there as well as the sync backup. This turned out to be useful for other reasons that I will get to below. The build really was a confluence of happy accidents!

There were two weaknesses that I saw in the build. One is how does the inbox processing know what is in the vault so as to link the new information to the correct places, the second is the same issue that led me to the Chat Miner, is all this information just falling into a void never to be seen again. This led me to step 3.

STEP 3: GRAPHS AND EMBEDDING

I run MIYO with my Copilot so my first instinct was to use embedding to link everything up. This did not really work. The embedding could put the notes on the map but it wasn’t able to do some of the things I needed. It was not able to link up arguments or make judgement calls apart from how similar the Inbox note was to others that already exist. Embedding was too shallow for what I wanted.

I think one of the strengths of llm is their ability to identify semantic similarity. They are stronger semantic elephants with an enormous memory than they are stochastic parrots reasoning without thinking. That led me to Graphify which is a different type of semantic analysis than embedding.

I freely admit that at this point I am at about the limit of my understanding of llm architecture so apologies to the techies for all my technical errors! At some level I am happy with the cybernetic approach and focus on the outcome without looking into the box too much. My understanding is that embedding is shallow and wide, whereas Graphs are deep and narrow. My thought was to combine them and they could balance each other out. Turns out it worked!

I put graphify (calling glm 5.3) and an embedding model (calling qwen3-embedding-8b) on the server, both back ended on TensorX, so that it propagated across all versions of my vault no matter the machine. It took a while to run on a 4GB memory box but it got there in the end. That makes it sounds like an easy ride. Embedding was easy, one run and done. Graphify was a litany of fuck ups and dead ends. This write up is already too long to document them here, I did keep a note in my vault with them all so maybe a dead end write up would be useful.

The llm might explain this better than I do:

LLM

  • The job. An inbox run turns a new item into a note, then answers one question: where does this belong in a vault of 1,000+ notes? Two systems on the server supply the raw material, working in opposite ways.
  • The map (embedding). Every note has a position on a giant similarity map, placed by how alike the words are. The new note gets a position in seconds, and the map returns its nearest neighbours. Complete, instant, cheap. It cannot follow an argument, and it cannot say why anything is similar.
  • The reading (graphify). A second system has read the whole corpus through an LLM, which recorded which notes genuinely relate — builds-on, argues-with — and which notes everything else routes through. It is the memory of the vault having been read. Deep, but slower to refresh, and a brand-new note is not in it until the next reading run.
  • The handoff. The run asks the map for neighbours, and the reading for the argument lines and load-bearing notes the map cannot see. It verifies every candidate against the notes on disk, picks 2–5, writes a reason for each, and patches the link in both directions. The map finds the candidates. The reading says which ones matter. Unticked chat-capture notes stay outside both systems — nothing unapproved is mapped or linked.
  • The record. Every run ends by writing one entry to a log note in the vault, newest at the top: what was processed, what was linked, what was renamed, what was left for next time. Any change traces back to the run that made it.
  • Why neither alone. Map only: it links to whatever shares vocabulary, and misses the note that argues the same point in different words. Reading only: it is a step behind, with no entry for the new note — it cannot rank anything from a note it has never seen. Each covers the other’s blind spot.

This solved my problem of hooking up my new notes to my existing corpus. It did it very well, I was happy.

The thing I wanted from all this was still missing though. Everything was being hooked up silently, llms on a server, recorded in my vault change log (which records every llm change from anywhere to my sorted notes) but it was not being surfaced for me to see. This is what led me to step 4.

STEP 4: HUBS ARE MY GARDEN

Previously I used MOCs before my hubs, dataviews, tagging notes, all that jazz to tend my knowledge garden. They were fine but again I had trouble keeping them current. Karpathy uses his wiki index. I honestly gave it a go but failed so I wanted to offload this to my llm helpers.

My idea was that with over 1000 notes in my vault there must be communities that would surface the first time I would do this but I wanted a living garden where communities could flourish or die without my intervention. My goal was that I could feed and prune the garden but how the hubs grow will be organic (or whatever the silicon version of organic is). I can harvest ideas from the hubs but always as new notes (which then go back into the embedding & graph).

I had embedding and graphs. Surely this was going to be easy. This was defintely as case of the engineers mantra of we do this not because it is easy but because we thought this would be easy.

The big issues included:

  • lack of hub identity so the near identical hubs spawned repeatedly.
  • Links in the hub fed into the Graph so the hubs themselves became gravity centres instead of reflections of the corpus. The bigger the hubs the more they pulled in notes, the bigger they got, repeated hourly! That one was a mess!
  • The clustering method either pulled in everything (700 hub notes) or atomised everything into thin hubs. One hub was just my notes on one book. Joining the dots in my vault was taken a bit too literally!

Working through this I ended up with the following metaphor: the embedding model prepares the beds in the garden and the graph plants the flowers. That worked for my non-techie brain. I am sure an actual engineer could explain this much better!

As always the llm explanation for the technical part is below:

LLM

Final build — engineering summary.

  • 1. Overview. The hub layer runs hourly as a pure function over two inputs: the graphify knowledge graph and the embedding index. Inputs in, hub notes out. No state is kept between runs; the hub note is the database.
  • 2. Inputs. Two independent lenses on the same corpus. Graphify: an LLM has read every note and recorded which genuinely relate — builds-on, argues-with — plus per-note degree centrality. Embeddings: qwen3-embedding-8b places every note in a 4,096-dimension vector space; a note’s position moves only when the note moves.
  • 3. Proposal (graph lens). Degree analysis identifies the load-bearing notes the rest of the corpus routes through. Each becomes a semantic kernel — a candidate hub.
  • 4. Membership (embedding lens). Members are all notes within cosine 0.65 of any kernel member, plus notes the graph links in with a weight the geometry cannot see. Every candidate is verified against the notes on disk.
  • 5. Identity. The kernel is written into the hub note’s frontmatter at mint and is immutable. The name is minted once. Membership is recomputed from scratch on every run.
  • 6. Merges. Hub families that drift within cosine 0.85 merge and re-mint from the union’s claims. The merged hub’s narrative is regenerated, never inherited.
  • 7. Kills. A median-anchored gate: a hub dies when its web × external flow falls below 40% of the live median. Kills are unconditional; everything killed survives in git and the snapshots.
  • 8. Change control. A corpus signature gates the hourly run. Unchanged corpus: one $0 ledger line and exit. Changed corpus: only affected views are rewritten; the third identical run writes nothing.
  • 9. Governance. Hubs are type/moc and excluded from the graph, so membership links create zero edges. The layer has no human interface: read the hubs, answer with new notes, never edit a hub. A stream log records every formation, merge, re-mint, and death.
  • 10. Output. 8 hubs, 10–73 members (median ~20), all canonical anchors of the corpus united in hubs for the first time. The count had no fixed cap; the median-anchored gate keeps it finite and rising only with the corpus’s genuine topic diversity.

This worked really well. It all sits on the server. It runs every hour, 30 minutes after the inbox run so there is time for the inbox to be processed before the hubs update. I also added in a summary section to the hub which sits at the top. It serves to orientate me when I open the hub to explore the topic.

I never edit the hubs, that is an llm job. What I do is open them up and start thinking about the notes, the shape of the information in there, and then I make my own notes as it sloshes around in my brain. The other thing that is happening is that I spotting gaps in the hubs, arguments that are half formed, or ideas that should be stress tested. When I spot a gap go do my own research

I added a hub section to my home page. It updates slowly as the hubs grow, shrink, and split. The llm spent much time telling me that I should not let hubs die automatically, I could see the point but it is not what I wanted to achieve. I do not have time to curate my vault so hubs would live forever if they did not fade away.

BONUS STEP - HUB VIZ

Keeping with the theme of unplanned happy accidents the git on the server became my hub tracker. It sees all the changes to the vault so I thought I could use it to vizualise the hubs over time. The stream is below. I can track when they get created, when they move rank, and eventually when they retire.

Graph View

Backlinks