The Knowledge Base Is Code: Keeping AI Support in Sync With a Product That Ships Daily

A user asked our support agent whether a certain template could handle large-scale SEO. The agent answered confidently, and wrongly. The model hadn't hallucinated; the knowledge base it read from had been written three product cycles earlier, before we'd documented the FARM-vs-Next.js distinction. The code had moved. The knowledge hadn't.

That gap is the problem this post is about. On Emergent, most user queries are answered by AI agents: one for escalations, one for live chat. Every one of them is only as good as the knowledge base behind it, and the product underneath that knowledge base ships every day. Features land, flows change, an edge case gets fixed. Each merged pull request makes the knowledge base a little more out of date.

For a while we did what everyone does: a human noticed the KB was stale, opened the CMS, and edited an article. It worked exactly as well as "a human remembers to" ever works. So we built a pipeline that watches the product change and proposes the knowledge to match, with a human in the loop and a git commit at the end.

One conviction kept it simple: the knowledge base is code.
‍

THE STAKES

What stale actually costs

It's tempting to treat documentation drift as cosmetic. In an AI-first support stack it isn't. The failure chain is short and expensive:

1. A PR changes real behavior: how a deployment cookie is scoped, or that .mjs files are now served with the right MIME type.
2. Nobody updates the KB, because updating the KB is nobody's job in particular.
3. A user hits the changed behavior and asks support about it.
4. The agent retrieves the old article and answers from it, confidently wrong.
5. The user escalates. A human is now doing the investigation the KB should have prevented, and the wrong answer is still sitting in the article, waiting for the next user.

Multiply that by a platform where millions of people build every day, and "we'll update the docs later" becomes one of the more expensive sentences in the company.

The insight that reframed the project: we already have a perfect, machine-readable signal for "the product just changed." The merge event. Every meaningful change to Emergent arrives as a pull request into one of a handful of core repositories. If a merge is the moment the knowledge base becomes stale, a merge is exactly the moment to ask whether it needs updating.
‍

CORE DECISION

The knowledge base is code

Before the pipeline could work, an earlier decision had to hold: our knowledge base is not a CMS. It's a git repository.

We'd already replaced our vendor-hosted KB with an in-house engine: markdown articles with YAML frontmatter, living in a repo, embedded into a vector index, served to agents over MCP. The markdown is the source of truth; the embeddings are a build artifact; the MCP server is stateless and reads the latest index.

That choice made everything downstream cheap. Because the KB is code, changes are pull requests: reviewed, auditable, revertible. "Update the KB" means "open a PR," the same primitive our engineers already use fifty times a day. And the reindex is a CI step: merge to the KB repo, embeddings rebuild, agents see the new content. No server restart, no CMS deploy.

So the pipeline doesn't need to be a knowledge base. It needs to turn "a product PR merged" into "a knowledge base PR, ready for a human to approve."
‍

THE SYSTEM

The loop

Two properties of this loop matter more than any individual box. First, it closes: a product change flows all the way to the answer a user gets, and every step in between is inspectable. Second, a human sits in the middle, on purpose. The pipeline never edits the knowledge base directly. It drafts; people decide.

Watching the right signal

The pipeline draws from three sources, but the primary one is merged pull requests. When a PR merges into a core product repository, a lightweight GitHub Action fires a webhook with the PR URL. That's the entire integration on the product side: no plugin per repo, no coupling to the pipeline's internals. A repo opts in with a small workflow file or it doesn't.

The gate is merged, not closed. GitHub sends the same event when a PR is merged and when it's abandoned; we only care about the former, so a single condition on the merge flag filters before anything is sent. Shipped code, and only shipped code, gets a look.

The other two sources round out the picture. When our support tooling flags a genuine knowledge gap on a ticket (a question the KB couldn't answer), that becomes an input; real user confusion is a first-class signal that the KB is missing something. And sometimes a human just knows an article is due and kicks off a run directly. Different triggers, one pipeline.
‍

FILTERING

Teaching a model to say "not relevant"

The naïve version of this system drowns you in noise. Most merged PRs are not knowledge-relevant: a refactor, a build-pipeline tweak, a library bump. If every merge produced a documentation suggestion, reviewers would tune it out and the whole thing would rot.

So the first real job of the pipeline is a gatekeeper: read the PR and decide whether it changes anything a user would ever need to know. An OG-card build-pipeline change gets dropped. A change to how auth-wall cookies are scoped goes through, because users will hit it. [DATA: what fraction of merged PRs clear the gate? Even a rough ratio, e.g. "roughly 1 in N merges", makes this section land.]

For PRs that clear the gate, the pipeline asks a sharper question than "should this be documented?" It asks what the knowledge base currently says about it, and classifies the delta:

• NEW: the KB has nothing on this; a new article or section is needed.
• INCOMPLETE: an article exists but is missing this; add to it, at a specific place.
• OUTDATED: the KB actively says something that is now wrong; correct it.

That last category matters most. A missing article is a gap a user works around. A wrong article is a trap the support agent walks into with full confidence. OUTDATED detections prevent confidently-wrong answers, and they're only possible because the pipeline reads the live KB before proposing anything.

Each recommendation carries where it should land (target article, section, insertion point), the proposed content, a confidence score, and the reasoning. Low-confidence suggestions are filtered out; near-duplicate suggestions get merged so a reviewer sees one clean proposal instead of five overlapping ones. The prompts driving each stage are editable without a redeploy, because prompt tuning is a daily activity, not a release event.
‍

THE TRADEOFF
‍

Why a human stays in the loop

We could have let the system commit its own changes. We deliberately don't, and it's worth being precise about why, because "keep a human in the loop" is easy to say and easy to turn into theater.

The knowledge base is the ground truth our support agents reason from. An error in it doesn't stay contained; it gets retrieved, cited, and repeated to every user who asks a related question until someone notices. The blast radius of a bad KB edit is larger than the blast radius of the PR that inspired it. That asymmetry is the entire argument: the cost of a wrong article is high, the cost of a human glance is low, so review wins.

Every recommendation lands in a review queue, routed by source repository, so the engineers closest to a change see the proposals it generated. A reviewer can approve as-is, edit and approve, or reject with a reason. Only on approval does the pipeline open a pull request on the KB repository. From there it's a normal PR: reviewable again, mergeable, revertible. When it merges, CI re-embeds the affected articles and the agents pick up the new answer immediately.

Rejections aren't wasted. A rejected suggestion is archived with its full context: a record of what the system proposed and why a human said no. Over time that's the raw material for making the pipeline itself better.

What it stores, and what it refuses to

The system keeps almost nothing the knowledge base already owns. It doesn't store article content; that lives in git. What it stores is proposals and decisions: each recommendation with its delta type, target, confidence, and status as it moves from pending to accepted, edited, merged, or rejected; each run with its progress log; reviewer routing; prompt versions; and the audit trail of what shipped versus what the model originally drafted.

That separation is the point. This is a proposal engine bolted onto a git-backed knowledge base, not a second copy of the truth. If it disappeared tomorrow, the knowledge base would be exactly as valid as it is today; you'd just be back to updating it by hand.

There's a piece of symmetry we didn't plan: the pipeline is itself built on Emergent. A FastAPI-plus-React-plus-Mongo app, generated and deployed on the platform it helps document. The tool that keeps our knowledge honest is dogfood.
‍

CONTEXT
‍

What about Google's Open Knowledge Format?

When Google Cloud published the Open Knowledge Format in mid-2026 (a vendor-neutral spec for representing organizational knowledge as a directory of markdown files with YAML frontmatter, meant to give AI agents curated context), we read it and recognized our own knowledge base. We'd landed on essentially the same shape before it had a name: markdown with structured frontmatter, organized in directories, versioned in git, readable by anyone who can cat a file. OKF's thesis, that the LLM-wiki should be just files, portable and diffable, with no proprietary account required to read or serve it, is the bet we'd already made when we moved off a hosted CMS.

So we see OKF as validation and a useful convergence point, not competition. But a format doesn't address the problem this pipeline exists for. A knowledge format assumes the knowledge is already written and just needs structuring. That's downstream of the hard question: is this still true, now that the product shipped again this morning? OKF gives you the container. Keeping what's inside the container true is a staying-current problem, and no format will do that for you.
‍

RESULTS
‍

What changed

The honest version of "results" is a change in default behavior. Before, keeping the KB current depended on a person remembering to. Now the default is inverted: every shipped change surfaces automatically as a question ("does the knowledge base need to move too?") with a drafted answer attached, waiting for a yes or no.

The concrete wins:

• The window between "the product changed" and "someone is checking whether the docs should" collapsed from "whenever someone notices" to "the next review pass." [DATA: median time from product-PR merge to KB proposal reviewed, even approximate.]
• OUTDATED catches stop confidently-wrong support answers before a user ever triggers them. [DATA: count of OUTDATED corrections merged since launch, or one concrete example of a caught wrong answer.]
• The reviewer's job got smaller. Instead of finding what's stale, the hard, invisible, easy-to-skip part, a reviewer approves, tweaks, or rejects a concrete proposal pointing at an exact article and section.
• Every KB change is a reviewable, revertible commit, with a paper trail from the product PR that motivated it to the answer a user eventually reads.

As with a lot of infrastructure we've built, the data model did most of the work. Once we accepted that the knowledge base is code (versioned, reviewed, reindexed by CI), keeping it in sync stopped being a documentation problem and became a pipeline problem. Watch the merges. Draft the delta. Ask a human. Open a PR.

A product that ships every day will always try to outrun its own documentation. The fix isn't writing faster; it's making the documentation move on the same signal the product does.

Emergent lets anyone build and ship production apps from a prompt, including, it turns out, the tools that keep our own platform honest. Start building on Emergent today.