Why Analytics Agents Need Decisions and Definitions — Not Just RAG

Point an AI agent at a data team’s history and ask it a definitional question — “how do we count active users?” — and retrieval will do its job well. It will find documents, dashboards, and threads that mention active users. What it will not do is tell you which of them is right. That gap is the difference between an agent that looks helpful and one you can trust to answer.

RAG retrieves text, not truth

Retrieval-augmented generation is a good tool for the problem it solves: finding relevant passages and letting a model answer from them. But a data team’s history is full of passages that are relevant and wrong — superseded definitions, deprecated dashboards, one-off analyses with a bespoke filter, decisions that were later reversed.

Retrieval ranks by relevance. Metadata, recency filters, freshness dates, and reranking can push a three-year-old dashboard down and this quarter’s agreed definition up — and a well-tuned pipeline should use them to reduce stale retrieval. But those signals rank documents; they do not, on their own, establish which definition the team has decided is authoritative. Ranking is not a governed decision. So the agent, handed several plausible definitions, still has no principled basis to choose the one the team stands behind — it picks one, states it confidently, and the analyst downstream has no signal that a choice was even made.

The definition problem

Years of analyses, definitions, decisions, dashboards, and code accumulate faster than any team can maintain their meaning. The result is that the definitions an analytics agent needs most are the ones most likely to be in quiet conflict. A few illustrative shapes of the problem:

• active_user counted over a 28-day window in the product’s own reporting and a 30-day window in finance’s — same name, two numbers.
• churn redefined in a v2 spec while three live dashboards still compute the v1 version.
• revenue cited everywhere, but with no source an agent can point to as the one it is allowed to use.

An agent running on retrieval alone will surface whichever of these it ranks highest. It cannot flag that the metric is contested, because “contested” is not a property of any single retrieved passage — it is a property of the relationship between them.

What an agent actually needs

To answer definitional questions safely, an agent needs three things retrieval does not supply on its own:

• Governed definitions. One current, agreed definition per metric, marked as current — distinct from the older versions still lying around.
• The decisions behind them. Not just the definition, but the fact that it was decided, and what it replaced, so a reversed definition does not resurface as live.
• Evidence and disagreement made visible. Each definition tied to its source, and genuine conflicts surfaced as conflicts, so the agent can say “these disagree” instead of silently choosing.

That is a maintained, governed source of truth over the team’s history — the decisions and definitions in force, the ones that were ruled out, and the places where sources still disagree — rather than a pile of text to retrieve from.

RAG as a component, not the answer

None of this means retrieval is wrong. It means retrieval is a component. It is how you find the candidate material; it is not how you decide which candidate is authoritative. The decision — which definition is current, which was ruled out, whether a number is safe for an agent to state — has to be governed, and ideally confirmed by a person, before an agent relies on it.

An analytics agent grounded only in retrieval will always be one stale dashboard away from a confident wrong answer. An agent grounded in maintained decisions and definitions knows the difference between a passage that mentions a metric and the definition the team actually stands behind. For serious data work, that difference is the whole job.