The first AI use case is organisational knowledge: lessons from building a governed RAG assistant

Most AI effort goes into how to build. The value sits in better decisions. A case for making governed knowledge accessible to leaders, with the guardrails, evaluation and honest failures of a working RAG prototype.

We love to talk about how to build AI. We spend far less time on what to build and why. I have sat in many of those rooms. The model gets the debate. The decision it should help us make gets ten minutes at the end.

I have come to believe the value is not in the model. It is in better, faster and more confident decisions. And those decisions depend on something simple that is rarely in place: knowing what the organisation already knows. Which KPI defines success? Is it approved? Who owns it? In most organisations the honest answer is “ask around”.

This has two sides, and I want to name both. Leaders and CXOs need that knowledge to be easy to reach, so they can scope projects faster and trust what gets delivered. And teams need to see their documents as the source other people will rely on. That changes the care they put into writing them. Access and diligence feed each other.

So I tested the idea. I built a small RAG assistant that answers only from governed documents. It cites every source, shows whether each one is Certified, Provisional or Deprecated, and says “not found” when the documents are silent. Then I ran it against 40 questions that I wrote before tuning anything. The failures taught me more than the successes, and I will show you both.

Everything here is synthetic data for a fictional retailer. This is a prototype and an experiment, not a production system. The code is on GitHub: kpi-governance-rag.

The problem: slow intake, scattered knowledge

Let me start with a picture most of us know.

Someone sends a request. It comes by email or chat, in whatever shape they chose. Now somebody has to find the KPI definition, track down its owner, check if the data is ready, and ask the same questions the last request also asked. A few days pass. Then two people give two different answers, because one read the approved document and the other remembered an older one.

That is what a slow intake process feels like. Good ideas wait in a queue for facts that already exist somewhere. Decisions get made on partial information. Leaders either wait, or decide without confidence.

I have seen the opposite too, and it changed how I think about this. Last year, tariffs suddenly became big news in the US. A Fortune 500 client’s data and analytics team wanted to move fast, because supply chain optimisation could save a lot of cost. But not everyone knew where to begin. What helped was a metadata catalogue I had built earlier. It linked each use case to its data products, each data product to its KPIs, and each KPI to standard, governed field-level mappings, all filterable by domain. The team did not start from zero. They started from what they already had.

On another engagement, a client needed to work on Net Sales. They had already built Gross Sales, but margins were a pain, and even Gross Sales had taken time in the beginning. Because every build had been logged with the right details, the Net Sales work gave a clear picture of where the organisation stood. Planning got easier. The lesson I took: when you know what you possess, you plan faster. That is what I wanted a small assistant to do for leaders.

The sharpest case is when two documents disagree. A certified definition says an Active Customer bought something in the last 30 days. An informal team FAQ says 90. Both look official enough to quote. So two dashboards show two numbers, and a meeting gets spent arguing about which one is right.

An assistant that answers from governed documents, with the source and status attached, turns that hunt into a question you can ask.

What should we call it?

I call it a governed knowledge assistant. Technically it is RAG, short for retrieval-augmented generation. It looks up passages, writes one answer from them, checks the citations and stops. It does not plan, use tools or act on its own, so it is not an agent. I am planning an intake agent next that would use this assistant as a tool. That is not built yet, so I will not claim it.

Why RAG, and not just a chatbot? A chatbot answers from what it learned in training. That can be old, generic or simply invented. RAG answers from your own approved documents and shows the source next to the answer. For governance, that difference is everything. You can check the answer, and you can see which document is thin.

Where the value shows up

I have not measured business impact, so please read this as design intent. These are the five places I expect value, and what in the prototype points to each.

1. Leaders’ time. A scoping question could take two meetings and three messages. It could become one question with a sourced answer.
2. Confidence. Every answer shows its source and status. A leader can see what is approved and what is still provisional before putting money behind it.
3. Diligence in the team. When the assistant says “not found” or “Provisional”, it shows exactly where the documents are thin. Now owners have a reason to write, review and certify, because people read their work through the assistant. Your documentation becomes the product.
4. Knowledge transfer. New joiners, and teams taking over work, can ask instead of hunting. What a leaver knew only helps if somebody wrote it down. The assistant makes the gaps visible.
5. AI use-case development. Every new AI idea needs the same facts first: which KPI, how certified, which owner, which data. The assistant gives them in seconds. That is the base for a faster intake process.

Here is the change I have in mind.

Intake process, today and with the assistant

The prototype covers the highlighted step: a sourced answer with its status. The gaps it exposes go back to the owners.

How I built it, and why each step is there

Think of the system as three loops. I prepare the documents when they change. I answer every question. And I measure every change I make.

System overview, 3 loops

The first loop is boring on purpose. I made 50 documents for a fictional retailer: KPI definitions, a catalog, procedures, policies, a glossary and an FAQ. Each one carries its owner, version and status. I put two traps in on purpose. One is a retired churn definition. The other is an informal FAQ that contradicts a certified rule. A governance tool that only works when the documents agree is not a governance tool. The documents are cut into 52 passages, turned into vectors with the bge-small model, and stored in Chroma. I tested three ways of cutting them and kept the simplest.

The second loop is the one users see. This is the path of one question.

Answer path, 4 steps, 1 check, 1 retry

The search finds the top 5 passages. Each passage reaches the model with its status label: Certified, Provisional or Deprecated. The model gets three rules. Answer only from these passages. Cite every claim. If the passages do not cover the question, say “not found”. Then plain code, not the model, checks that every cited source was one of the five. If not, one retry, and then a refusal. The model does not mark its own homework.

Every call also leaves a trace: the question, the sources, the tokens and the time. That is your audit trail.

The answer model is Gemini 3.5 Flash-Lite on the free tier, at temperature 0. A small Streamlit app shows the answer, the numbered sources and their status.

Why I wrote the test questions first

A demo shows you the questions that work. A test shows you the ones that do not. So I wrote the test before I tuned anything.

There are 40 questions. Thirty are for development, where I try changes again and again. Ten are held back and run once per version, so I cannot bend the system to fit them. Eight questions have no answer in the documents, because refusing correctly is part of the job. Some are traps, like the 30 against 90 days rule. Some are bare KPI codes, typos and short forms.

Golden set, 40 questions, 10 held out

Fifteen of the questions I wrote myself, after reading a few documents and before running any search. The other 25 were drafted with Claude and reviewed by me. Each question lists the key facts a correct answer must contain, and each fact is scored yes or no. I also wrote the pass rule for any prompt change before I saw the held-back result. That sounds strict. It saved me from fooling myself.

For a business, this set of questions is a contract. It says what the assistant is promised to answer, and it gives the data team and the business one shared way to say better or worse. Without it, “the AI works” is only an opinion.

What the numbers say

Search first. Recall@5 is the share of the needed documents found in the top 5. Hit@5 asks if at least one needed document is there. MRR@10 rewards putting the first right document high.

• Dev (24 answerable questions): recall@5 0.815, hit@5 1.000, MRR@10 0.938.
• Holdout (8 answerable questions): recall@5 0.729, hit@5 0.875, MRR@10 0.646.

Now the answers. Key-fact coverage is the share of required facts the answer states. Faithful means no claim goes beyond the retrieved passages. Version 1 is my default prompt. Version 2 adds one rule that tells the model to state every key fact.

• Key-fact coverage: dev v1 75.6% (59 of 78), dev v2 79.5% (62 of 78), holdout v1 60.9% (14 of 23), holdout v2 65.2% (15 of 23).
• Answers with every key fact: dev v1 16 of 24, dev v2 18 of 24, holdout v1 3 of 8, holdout v2 4 of 8.
• Faithful answers: dev v1 22 of 22, dev v2 22 of 22, holdout v1 8 of 8, holdout v2 7 of 8.
• Unsupported claims: 0, 0, 0 and 1 (the one is holdout v2).
• Wrong refusals of answerable questions: 2 on dev (both versions), 0 on holdout.
• Unanswerable questions refused correctly: 6 of 6 on dev, 2 of 2 on holdout, both versions.

One answer costs about $0.0008. That is a calculation from a public price page, because my runs were on the free tier. On the dev set the model call took a median of 4.6 seconds, and 8.4 seconds at the 95th percentile. Holdout timing was too noisy on the free tier to report.

Five lessons from building it

I lost a few small battles while building this. Each one taught me something I would not have learned from a win.

Get Jaishri Rai’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

1. The simple search won. I tested search by meaning, search by keywords, a mix of both (called hybrid), and a second-pass ranker. The mix lost to plain search by meaning on dev, in all three ways of cutting the documents (recall@5 down 0.05 to 0.10). The ranker was mixed on dev (recall +0.010, MRR@10 -0.028) and better on the 8-question holdout, but about 50 times slower. So I kept the simple one. Lesson: measure before you buy complexity.

2. Good search is not the same as authority. Search finds relevant text. It has no idea which source is approved. The ranker even pulled the informal FAQ into the top 5 on 11 dev questions, instead of 7. That is why the status label and the prompt rules sit in the answer step. On the trap question, the answer gives the certified 30-day rule and calls 90 days informal.

3. My own scoring had a bug. I used an AI judge to score the answers and checked it against my own hand grades. The first check missed my bar: 80.6% agreement, when I had asked for 85%. All six misses came from one question where my key facts asked for job titles and the question asked for teams. I fixed it in the open, and agreement went to 100% on 31 key facts. I kept both numbers in the write-up. A scoring mistake can pick the winner, and I would rather you see it.

4. The smarter prompt lost. Version 2 told the model to state every key fact. It covered 3 more facts on dev and 1 more on the holdout. I wanted it to win, honestly. But on the holdout it named an owner that the retrieved passages never stated. It had guessed from a nearby action owner, and the guess happened to be true. My pass rule, written before the run, failed on faithfulness, so I kept version 1. In a governance tool, a complete answer that is partly guessed is the wrong trade.

5. Most of what is left is search. Two broad list questions were wrongly refused, because only some of the needed documents were found. A question that names a KPI only by its code pulled process documents instead of the KPI. Hybrid search did not fix that either.

Where it fails, and the guardrails around it

Read this part before you trust any of it.

• Holdout key-fact coverage is 61%. Almost four in ten required facts were missing.
• The holdout has only 8 answerable questions. One question moves a score by about 4 percentage points. These results show direction, not proof.
• The corpus is synthetic and small, 52 passages. Search results here will not carry over to a large corpus.
• The judge is the same model that wrote the answers. I checked it on only 10 answers, and only 2 of the 31 human grades were “no”. So a lenient judge is weakly tested.
• The citation retry has never fired on a real answer. I tested it with forced failures only.
• Temperature 0 is not fully repeatable. The same question gave different citations on different days.
• The cost is a calculation, not a bill.

Now the guardrails. Governance is easy to say and hard to show, so for each risk here is what I built and what evidence I have, with its limit.

• An invented answer. Guardrail: answer only from retrieved passages, and refuse when they do not cover the question. Evidence: 8 of 8 unanswerable questions refused, and every non-refused answer judged faithful except one v2 answer. Limit: same-model judge.
• An outdated or informal source wins. Guardrail: a status label on every passage, plus prompt rules. Evidence: the trap question gave the certified 30-day rule, and the label is shown to the user too.
• A fake citation. Guardrail: code checks that each cited source was retrieved, with one retry and then a refusal. Evidence: tested with forced failures, never triggered by a real answer.
• A prompt change makes things worse. Guardrail: a pass rule written before the holdout run, and the holdout run once per version. Evidence: version 2 was rejected on faithfulness.
• Biased scoring. Guardrail: the judge was checked against 10 answers I graded blind. Evidence: 100% agreement after a logged fix. Limit: a weak test for leniency.
• No audit trail. Guardrail: one trace line per call with question, sources, tokens and time. Evidence: every call is logged.

Two more pictures from the app show the guardrails at work. The first is a governance rule. Provisional KPIs may not go on the Executive pack, and the answer carries a yellow warning because it used a source that is not Certified. The second is the refusal. When the documents say nothing, the assistant does not guess.

What I measured, what I did not, and what I take away

Let me be clear about the edges. This is a prototype on synthetic data. I did not measure hours saved, faster approvals or better decisions. I have not run it with real leaders or a real intake queue. If someone tells you a prototype like this “proves” business impact, be careful. It does not. I will not say it either.

What I did measure is narrower, and I think more useful: does the assistant answer from the right document, does it stay faithful to it, and does it say “not found” when the answer is not there. On those three, I have numbers, and I have failures to show next to them.

Here is what I take away.

1. Decide what to build before you decide how. The stack took days. Knowing which questions leaders actually ask took longer, and mattered more.
2. Know what you possess. The assistant is only as good as the documents behind it. Weak definitions in, weak answers out. That is a data strategy problem, not a model problem.
3. Write the test questions first. Without them, “it seems to work” is the only evidence you have. With them, you can say what improved and what did not.
4. Put the guardrails in code, not in hope. Every answer must cite a source, and the source must exist. If not, one retry, then a refusal.
5. Documentation becomes a team habit. When people know their documents are what the assistant will quote, they write them with more care. I did not measure this. I believe it, and I would test it first.

The assistant does not replace the leader’s judgement. It gives the leader a faster, safer way to reach what the organisation already knows.

Build it yourself

If you want to build the same thing, everything is open. The code, the 40 test questions, the results and the screenshots are in the repo: github.com/jaishrirai/kpi-governance-rag.

The stack is small, and all of it is free tier:

• Language and app: Python, Streamlit
• Vector store: Chroma
• Embeddings: BAAI/bge-small-en-v1.5
• Answer and judge model: Gemini 3.5 Flash-Lite, temperature 0
• Checks: citation validation in code, a trace line per call, 248 tests

The order I would follow:

1. Collect the documents and label each one: Certified, Provisional or Deprecated.
2. Split them into passages (mine gave 52).
3. Write the test questions, including questions that must be refused.
4. Embed, store and retrieve the top 5 passages.
5. Answer only from those passages, with a numbered source on every claim.
6. Check the citation in code. One retry, then a refusal.
7. Measure on the development questions. Touch the holdout questions once.

Start small. Your first version will be wrong in ways you cannot guess today. That is fine. The test questions will tell you where.

If you are building something similar, I would love to hear what broke for you.