Anthropic Got Its Analytics Agent to ~95% Accuracy. The Hard Part Wasn't SQL.

In June, Anthropic published a useful look inside its own analytics operation.

The headline numbers are impressive: Anthropic says 95% of its business analytics queries are automated through Claude, at roughly 95% aggregate accuracy.

The interesting part is what Anthropic had to build around the model to get there.

Without the Skills its data team created, Claude’s accuracy did not exceed 21% on Anthropic’s evaluations. With those Skills, Anthropic says accuracy consistently exceeded 95% in aggregate.

Then came the part every company building an analytics agent should pay attention to:

without active maintenance, Anthropic watched offline accuracy drift from roughly 95% to roughly 65% in one month.

This is Anthropic’s result on Anthropic’s own analytics system—not a Propperly benchmark, and not a promise that another company will see the same numbers.

But it exposes an important enterprise AI problem.

A capable model is not the same thing as a reliable analytics system.

The problem wasn’t writing SQL

It is tempting to think of an AI analytics agent as a better text-to-SQL interface.

Ask a business question.
Find a table.
Write the query.
Return the answer.

That is the easy version of the problem.

Anthropic describes analytics accuracy primarily as a context and verification problem. In its experience, most wrong answers came from three places:

Ambiguity.
The business concept maps to several plausible fields, tables or definitions.

Staleness.
The data model, business definition or agent documentation changed, but the agent’s understanding did not.

Retrieval failure.
The correct information exists, but the agent does not reliably find or use it.

Consider a simple question:

“How many active customers did we have last month?”

Your warehouse may contain several reasonable models.

A dashboard may use a 30-day definition.

A recent decision may have changed it to 28 days.

Three dashboards may still be waiting for migration.

An older notebook may encode an exception nobody remembers.

Retrieval can find all of those things.

The hard question remains:

What should the agent actually trust?

This is why giving an agent access to more company data is not enough.

What Anthropic had to build around the agent

Anthropic calls its approach an agentic analytics stack.

Reading that stack operationally, there are several distinct jobs that have to happen around the model.

1. Reduce the number of plausible truths

Anthropic emphasizes canonical datasets, governed models and strong metadata.

If “revenue” can resolve to forty plausible implementations, the agent begins in a bad position.

A reliable system tries to make the correct route explicit before the agent starts reasoning.

2. Give the agent sources of truth

The model needs more than tables.

Anthropic describes a hierarchy that includes its semantic layer, lineage and transformation graph, structured reference material, and business context such as documentation and decision logs.

The goal is not simply to make information searchable.

It is to help the agent know which information should govern the answer.

3. Encode domain knowledge

Anthropic uses a Knowledge Skill to route Claude toward the right domain references, tables, columns, joins and known pitfalls.

This narrows a huge retrieval problem into a smaller, curated one.

4. Encode how a strong analyst works

Anthropic also describes a Runbook Skill.

It captures procedural knowledge: clarify the question, choose sources, execute the analysis, validate the result and use reusable analytical patterns where appropriate.

That distinction matters.

An agent needs both:

what the organization knows, and
how a good analyst works with that knowledge.

5. Evaluate the system against real questions

Anthropic maintains offline evaluations based on representative business questions.

The point is straightforward: if you are building an analytics agent but cannot systematically measure whether its answers are correct, you do not really know what you built.

6. Keep the context synchronized with reality

This may be the most commercially important lesson.

Anthropic says its Skill documentation describes a data model that changes daily. When the documentation was not actively maintained, offline accuracy fell from roughly 95% to 65% over a month.

They eventually treated Skill maintenance as an engineering problem.

Anthropic reports that roughly 90% of its data-model pull requests now include a Skill change in the same diff.

That is not “documentation hygiene” — it is part of the production system.

Building the context is only half the job

This creates an awkward problem for companies adopting enterprise analytics agents.

You may already have the warehouse.

You may already have dbt, dashboards, a semantic layer, SQL, notebooks and documentation.

You may already have Claude or another capable agent.

But the context that makes those pieces usable is often scattered across years of analytical work.

Definitions live in documentation.

Exceptions live in old analyses.

Decisions live in tickets and conversations.

The current implementation does not always match the current decision.

And the people who know which source is trustworthy are exactly the people you are trying not to interrupt for every routine question.

So the real implementation question becomes:

Who is going to reconstruct, govern and maintain the layer between your existing analytics stack and the agent?

You should be able to test that before a large integration project

A company should not need to spend weeks with engineers, migrate its warehouse or build production connectors just to find out whether this approach is useful.

Start smaller.

Take one bounded analytics domain and the work that already exists around it:

data models, definitions, dashboards, SQL, analyses, documentation and decisions.

Then ask whether you can reconstruct enough reviewed context to answer a few concrete questions:

• Which sources are authoritative?
• Which definitions are current?
• What has been superseded?
• Where do sources genuinely disagree?
• What is still unresolved?
• What evidence should an agent show before a human trusts the answer?

If that reconstruction is not useful, stop.

If it is, then decide how much infrastructure is worth building around it.

That is a much cheaper question to answer than “How should we transform our analytics organization for AI?”

From RAG to current state

We have written before about why analytics agents need decisions and definitions—not just RAG.

The distinction becomes even more important here.

RAG can retrieve a definition that was correct six months ago.

A semantic layer can encode the current implementation.

A decision log can show that the implementation is supposed to change.

A ticket can show that the migration is incomplete.

All four can be accurate records.

Only one may be safe for the agent to use for a particular question today.

That is not just a retrieval problem.

It is a state problem.

The system needs to preserve what is current, what was superseded, what is disputed, what is unresolved, and what evidence supports each state.

And because the company changes, that representation has to change too.

The enterprise analytics agent is bigger than the model

Anthropic’s results are interesting because they make something visible that is easy to miss in a demo.

The model may be the most impressive component.

But reliable analytics depends on the less glamorous machinery around it:

governed sources
business definitions
current decisions
agent knowledge
analyst workflow
evaluation
maintenance

If every company has to rediscover and hand-build that layer independently, deploying reliable analytics agents will remain much more expensive than the demo suggests.

We think that layer should become a product.

At Propperly, we’re starting with Analytics: reconstruct the context from work the company already has, put humans in control of what becomes current, test it on a bounded domain, and then keep it aligned as the underlying work changes.

Building an analytics agent? See how we build the context foundation for a reliable analytics agent.

Prefer to prove the reconstruction before integrating anything? Run it yourself →

Source

The Anthropic figures and architectural observations in this article are based on:

Anthropic — “How Anthropic enables self-service data analytics with Claude” — June 3, 2026

Source URL: https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude

Specific claims used:

• 95% of Anthropic business analytics queries automated via Claude.
• ~95% aggregate reported accuracy.
• Without Skills, accuracy did not exceed 21% on Anthropic’s evals.
• Adding Skills consistently took aggregate accuracy above 95%, with some domains regularly around 99%.
• Anthropic identified concept/entity ambiguity, data staleness and retrieval failure as the dominant error classes.
• Anthropic observed offline accuracy drift from ~95% to ~65% over a month without active Skill maintenance.
• Roughly 90% of Anthropic data-model PRs now include a Skill change in the same diff.
• Anthropic treats canonical data foundations, sources of truth, Skills, validation and active maintenance as parts of its analytics-agent stack.