Why your analytics agent was accurate at launch and wrong six months later

An analytics agent built on today's metric definitions will give wrong answers once those definitions change. The problem is not the model.

An analytics agent gets deployed. The team runs it through a set of test questions and it performs well. Six months later, the same questions get different answers. The model has not changed. The underlying data has not changed. But the answers are wrong.

The cause is almost always the same: the definitions, decisions and scope rules the agent was built on have changed, and nobody updated the agent’s grounding.


What accurate means, and when it stops being true

When Anthropic published research on AI analytics agents, they found a roughly 95% accuracy rate on well-scoped questions with Anthropic’s own Skill documentation describing the data model. That number is real and meaningful. It is also a snapshot.

The accuracy rate was measured at a point in time, against a set of definitions that were current then. The 95% holds as long as the definitions hold. When they change, accuracy drifts, and the agent does not know to be uncertain.

This is not a failure of the model. It is a structural problem with how organizational knowledge is maintained, or not maintained, underneath AI agents.


Three ways definitions change after deployment

Analytics definitions are not stable. Here is what actually happens in the months after an agent goes live.

The definition gets narrowed or broadened. “Active user” was rolling 30 days at launch. The product team decides 28 days better reflects real engagement. The finance model still uses 30 days. The agent, if it was trained on the 30-day definition, now gives the old answer with full confidence.

A decision gets superseded without being recorded anywhere findable. In June, the analytics lead decides that churned accounts should no longer count toward the monthly active figure. This decision happens in a meeting, gets noted in a Slack thread, and propagates inconsistently across dashboards. The agent has no mechanism for learning that a decision was made, let alone what it was or why.

A conflict arises that nobody has resolved. The data warehouse team and the product team disagree about how to handle reactivated users. Both definitions exist. The agent either picks one without knowing it is contested, or gives an average that nobody intended, or refuses to answer. None of these outcomes is useful.

These three situations, a changed definition, a superseded decision and an unresolved conflict, look identical from the outside. The agent gives an answer that differs from what someone expects. The agent does not volunteer that the underlying state has changed.


Why the usual fixes do not work

The standard prescription is to update the semantic layer and retrain the agent. That is correct as far as it goes. The problem is what gets lost between the decision and the update.

The decision to change a definition happens when it happens. The semantic layer update happens when someone gets to it, if they know the decision was made. The gap between the two is a window in which the agent gives answers based on definitions that are no longer what the organization intended.

More importantly, the old definition is simply replaced. The agent now gives the new answer, but has no record of what the old answer was, when it changed, or why. A question like “why did our Q1 active user count drop by 12%?” can only be answered correctly if you know that the definition changed mid-quarter and that the old and new counts are not directly comparable. A system that overwrites without preserving the history cannot answer that question.

Documentation helps but does not fully solve this. Documentation gets created, falls out of date, and lives in places the agent cannot read or verify. The fundamental issue is that a change in organizational understanding needs to be represented somewhere that both humans and agents can access reliably, with its history intact.


The four states that actually matter

There is a more precise way to describe the maintenance problem. At any given moment, an analytics definition or decision can be in one of four states:

Current. This is the authoritative definition as of now. It is grounded in evidence and was formally accepted by the right person.

Superseded. This was the authoritative definition until a specific point. It is preserved, with the evidence that was replaced and the human decision that changed it. You can still ask what it said and when.

In conflict. Two or more sources make incompatible claims about this definition. The conflict has not been resolved. An agent asked to answer a question that depends on this definition should surface the conflict, not pick a side.

Unresolved. A decision needs to be made but has not been. This is different from a conflict: a conflict has two competing answers; an unresolved definition is an open question. Neither a confident answer nor a forced choice is appropriate.

An agent with access to this four-state representation can answer differently in each case: give the current definition confidently, note when supersession is relevant to a historical comparison, flag a conflict rather than conceal it, and decline to answer questions that depend on something unresolved.

An agent without this representation can only give the most recent answer it knows, with no indication of which state the underlying definition is in.


Why this matters more as organizations use AI more

As analytics teams deploy more AI, the problem compounds. An agent built in month one is used to answer questions in month six. The questions in month six often span the period in which definitions changed. The agent, lacking a record of when the definition changed and what it said before, either gives the new answer for the whole period (wrong for the earlier dates) or gives the old answer for the whole period (wrong for the later dates).

This is the core of why sustained accuracy requires something beyond a point-in-time semantic layer update. It requires a maintained record of how the organization’s understanding has evolved: what was decided, when, by whom, on what evidence, and what it replaced.

This is also why the problem shows up most clearly in analytics. Analytics organizations make explicit decisions about definitions all the time. They have to. Revenue, activity, churn, conversion: these concepts mean different things in different contexts, and the organization’s decisions about which meaning to use are real organizational outputs, not just configuration choices. Maintaining those decisions, with their history, is the maintenance job that makes analytics AI reliable over time.


What changes the picture

The fix is to maintain the organization’s analytical conclusions the way the organization actually uses them: as a governed, attributed current state with its history preserved.

This means conclusions are not stored as a flat definition file. They are stored with their standing (current, superseded, conflicted, unresolved), their evidence, the human decision behind each change, and the history of what they replaced. When a definition changes, the change is a new entry in that record, not an overwrite.

An agent drawing on this maintained state gets the right answer because it knows what is currently authoritative, can flag when a question spans a period when the definition was different, and can surface conflicts rather than silently picking a side.

Maintaining this record is not an automatic process. Someone needs to review and approve each change. That is the point: the organization’s decisions about what is authoritative should be made by the organization, not derived from the most recent model output or the latest database entry. The maintenance job is inherently human, and the infrastructure exists to make it tractable at scale.

If you are building analytics AI for the long term, the question is not only how accurate your agent is today. It is how you will keep it accurate as the organization’s definitions move.


Propperly maintains what your analytics organization has concluded, as a governed current state with its history. See how it works for analytics teams.

See how Propperly keeps your analytics agent current as definitions change.

Start with a bounded reconstruction of one domain using your own definitions, decisions, and dashboards.