LLM-Wiki as a Milestone Compiler for Agent Projects

Agents operating on codebases run into the same wall: context collapses after a milestone. Teams rely on ephemeral files like plan.md for in-flight work, but once the sprint ends, that memory is buried or lost. A compiled wiki solves this by forcing a durable synthesis of decisions, facts, and architecture at gate intervals. The result is a structured, typed knowledge base that reduces query-time token costs and prevents agents from re-deriving knowledge on every turn. Here is how to build the pattern, where it breaks at scale, and the infrastructure tradeoffs that keep it honest.

The Pattern #

The compiled wiki starts with a schema, not a dump. It structures knowledge into typed subdirectories—architecture/, reference/, features/, patterns/, testing/, session-log/—each governed by YAML frontmatter fields like date, type, status, and sources. An index.md at the root acts as the agent’s routing table, not decoration. Agents drill these index links to gather context before answering, and the schema tells the model exactly where to file new information and what conventions to follow.

A page is just markdown with a typed header. One of ours, in full at the top:

---
date: 2026-07-02
type: patterns
status: active
sources: plan.md M1.3, M3.8, harness/writer.py
---

# Output Hygiene: Never Trust Raw LLM Text as the Final Artifact

Four fields do the work. type decides which directory it belongs to and which sections it must carry. status lets an audit demote a page to stale without deleting it. sources is the audit trail back to the plan entries and code that justify the page — the thing that makes a claim checkable later. Nothing here is exotic; the discipline is that every page has it.

In our homelab, docs/wiki/*.md serves as a write-through cache: files are written locally first, preserving git history and surviving server outages, while the server synchronizes in the background. Crucially, these pages are compiled at milestone gates rather than continuously. When a milestone completes, a deliberate ceremony runs in order: test gate, wiki compile, planning, and delivery. This decouples the wiki from transient thoughts; it stays in sync with released decisions, not every experiment that failed or every false start. The structure is the mechanism that keeps the knowledge usable.

Wiki as Milestone Compiler #

Most engineering teams only capture the ephemeral half of their memory: a plan.md that tracks current tasks but disappears into backlog churn. A milestone compiler forces the durable half. In-flight decisions live in ephemeral files, and session handoff relies on memory.md, but at the gate, the agent must synthesize these into typed pages. This division of labor means the wiki captures what survives the sprint.

The friction of compilation is the feature. Without it, architectural decisions merge with operational noise. That pressure forced a structural split on us: reference/ had to be extracted from architecture/. Infrastructure work produces vast amounts of current-state facts—inventory tables, allocation maps, service diagrams—that describe what is true, not why a choice was made. If these stay in the architecture directory, the decision record buries itself under operational dumps. The rule is mechanical: if a page argues a choice with a rationale, it goes to architecture/; if it states a current fact with no “why,” it goes to reference/. This discipline ensures the compiled output is actually useful for future agents, forcing the model to distinguish between operational status and engineering decisions.

Structured Drill vs. Raw RAG #

The economic argument for a compiled wiki is cost alignment and reasoning quality. Karpathy’s LLM Wiki pattern formalizes the shift from retrieving chunks to navigating a compiled graph. Raw RAG re-derives reasoning on every query: the LLM retrieves context chunks and reconstructs the answer from scratch. A compiled wiki does the synthesis once, at write time. Subsequent queries navigate existing cross-references and contradictions. The compounding benefit is structural; as the wiki grows, cross-references accumulate and contradictions get flagged during synthesis. RAG injects static chunks; the wiki presents a curated view that has already resolved conflicts.

Be careful about claiming this saves money. Compiling is real work — the synthesis pass costs tokens up front, and it recurs at every gate. What changes is not the total spend but where it lands and what it buys: you pay once, at write time, for reasoning that a retrieval system redoes on every question, and the artifact you get back is one a human can also read and correct. The win is compounding quality, not a cheaper invoice.

RAG remains the right tool for large, unstructured document streams where no natural compile point exists — continuous logs, legal discovery, or a pile of notes with no milestone structure to compile against.

What Breaks at Scale #

As knowledge scales beyond a single repository, the wiki exposes failure modes that raw file trees hide. First, an agent in repo A cannot see that repo B solved the same problem, so decisions re-derive and quietly diverge. Second, vocabulary drift occurs: the same concept acquires a different name in each repo, killing cross-repo search. Third, promotion ambiguity arises. Without a mechanism for “is this local or global?”, teams either over-share—where a hacky workaround for a debug session in repo A gets indexed as a recommendation—or under-share, where everyone works in the dark. Finally, search collisions degrade relevance. Our own store co-hosts a non-wiki article corpus (ai_research, ~1,570 docs) alongside the wikis. Without a strict prefix filter, wiki results interleave with unrelated articles, burying the structured knowledge.

The shape of the solution is namespace isolation. Each repo gets a dedicated namespace (slug = repo name). A deliberately empty wiki-shared namespace exists for opt-in promotion. The agent must filter searches to a namespace or a wiki- prefix. Promotion to the shared namespace is manual—never automatic. A human must request it, or an audit must find a near-duplicate and ask. Automation would silently elevate one repo’s local decision to a global standard, polluting the shared view with local heuristics. The system should handle namespaces spanning an order of magnitude in size—wiki-proxmox at 137 docs down to wiki-job-search at 10—to prove the pattern works for small repos without over-engineering the infrastructure.

Infrastructure Mechanics #

The mechanics of the wiki require separating canonical state from search indexing. A namespace is a row in a namespaces table, carrying its own meili_index_uid and per-namespace ranking_rules_json. Documents live in a documents table partitioned by namespace_id, holding canonical_url, content_hash, body_markdown, source_repo, and written_by. Postgres is the canonical store; Meilisearch is a derived index, one per namespace. Keeping them separate is a deliberate choice rather than an accident of history: Postgres owns correctness and transactions, Meilisearch owns ranked retrieval, and neither is asked to do the other’s job well.

Synchronization uses a transactional outbox pattern: a write inserts the document and an index_outbox row in a single Postgres transaction, and a worker drains the outbox into Meilisearch. This guarantees atomic writes and prevents dual-write inconsistency. The cost is a ~2 second lag in search; reads on Postgres are immediately accurate, while the search index trails by the queue duration. This lag is by design, not a bug. content_hash on the document row enables compare-and-swap (CAS). The client stores the returned hash in a gitignored .sync-state.json; a stale hash triggers a conflict with the current server content, forcing a merge rather than a silent overwrite. The system sits behind a single MCP server exposing tools like ingest_document, search_namespace, and list_namespaces, ensuring the agent interacts with a unified interface regardless of the underlying schema.

Link Hygiene and Graph Integrity #

A compiled wiki is only as reliable as its graph. Cross-references via [[wiki-links]] require typed subdirectory prefixes—[[architecture/deployment-strategy]] rather than a bare [[deployment]]. Without this, two pages with the same filename in different directories collide, and the agent cannot disambiguate the target. Furthermore, broken links silently degrade retrieval. If an agent follows a dead link, it either fails to find context or fabricates a response.

Link validation belongs at compile time. When the agent compiles a milestone gate, it checks links during the write pass. A broken link is caught immediately, not weeks later when a new agent navigates the graph. This discipline prevents the wiki from decaying into a bag of unlinked documents. The agent performs the bookkeeping—summarizing, cross-referencing, and filing—but the schema enforces the structure. If the graph breaks, the agent falls back to raw RAG on the files, defeating the cost and reasoning benefits of the compiled approach. Typed prefixes and a maintained index.md keep the retrieval graph intact.

When Not to Use This #

The compiled wiki pattern adds overhead: defining frontmatter, filing into typed directories, maintaining the index, and running compile passes. This overhead only pays back at multi-repo scale, or when knowledge accumulates across sprints and must survive team turnover. A repo with five throwaway scripts, a single spike project, or a one-repo hobby side project does not benefit. For these cases, a simple notes folder or raw RAG is sufficient. The pattern requires a compile point; without milestones, gates, or at least a natural phase boundary, you have nothing to compile against, and you’re left with just another directory of notes.

Even at scale, the pattern has limits. A repo rename orphans its namespace — two of ours still carry pre-rename slugs. Cross-namespace relevance ranking remains unreliable—agents should judge snippet relevance rather than trusting score order. Most critically, the real failure mode is staleness. Pages are model-generated, so they drift from the code they describe — a renamed module lingers in prose and links long after the rename, and nothing in the write path notices. The content_hash prevents overwrite conflicts; it cannot detect a page that is merely wrong. The maintenance burden falls on the team to run periodic audits, comparing the wiki’s assertions against the actual codebase state. If you cannot commit to this review cycle, the compiled wiki will become a liability, sending agents down paths that no longer exist. Adopt this only when you have multi-repo scale and the discipline to audit the output regularly.