Why AI Analytics Still Gives Different Answers to the Same Business Question

We have cloud warehouses, transformation frameworks, catalogs, lineage, semantic layers, BI platforms, RAG, text-to-SQL, MCP servers, AI agents, and models that can reason over enormous amounts of information.

And yet a business user can still ask:

What is a new user?

and get three plausible answers.

Or:

What was revenue last quarter?

and receive a precise number that Finance does not recognize.

This looks like an AI hallucination problem. Often it is not.

The deeper problem is that the organization itself contains multiple pieces of evidence about what the answer means—and the AI has no reliable way to know which one currently governs.

More data access does not automatically produce more trustworthy answers.

The paradox: the model can find everything and still be wrong

Consider a fairly normal analytics environment.

The warehouse contains users, accounts, subscriptions, and events. The catalog has descriptions. The semantic layer has an active_customer metric. A dashboard has a retention calculation. A product spec says the definition changed in May. A ticket limits the migration to new reporting. Finance has a separate historical policy. An analyst recorded an exception in a notebook.

None is necessarily false. The question is: which applies now, to this user, for this purpose? That is an organizational-context problem, not merely a retrieval problem.

Schemas tell AI what exists, not what the business means

A database schema can tell an agent that created_at is a timestamp and account_id is a key.

It cannot reliably tell the agent:

• whether a “new user” means account creation or first meaningful activity;
• whether internal employees are excluded;
• whether a reactivated account counts as new;
• whether the definition changed last quarter;
• whether Finance and Product intentionally use different versions.

This is why enterprise text-to-SQL is much harder than demo text-to-SQL.

Spider 2.0 evaluates real-world enterprise workflows with complex databases, often containing more than 1,000 columns. Its original code-agent baseline solved 21.3% of tasks, versus 91.2% on Spider 1.0.

A newer benchmark, EntSQL, goes directly at another missing ingredient: proprietary business knowledge. Its examples require internal metrics, reporting conventions, and organizational rules beyond the question and schema. The best evaluated English system reached only 15.9% when long-form enterprise documents were provided.

The correct query often depends on knowledge that SQL schemas do not contain.

Metadata helps. It does not settle every question.

Better metadata can improve AI analytics substantially. Atlan reported a controlled benchmark in which governed metadata improved AI SQL accuracy by 38% relative to its baseline. Microsoft recommends descriptions, instructions, verified answers, and curated AI data schemas; Google grounds Looker conversational analytics in its semantic model.

But “more metadata” can still leave a critical ambiguity:

What is authoritative?

Suppose two documents describe net_revenue: one is newer, one was approved by Finance, one is scoped to a business unit, and an older version is still implemented despite a later decision to change it.

Retrieval can rank them, a catalog can describe them, and a model can summarize them. None automatically establishes which interpretation the organization stands behind.

A semantic layer solves meaning at the data-model level

For AI analytics, a governed semantic layer is becoming foundational.

Instead of asking a model to infer joins and metrics from raw tables, the semantic layer can provide defined measures, dimensions, relationships, grain, synonyms, and access rules.

Microsoft now tells Power BI teams to prepare semantic models specifically for AI consumption and warns that poor model design can lead to incorrect or misleading Copilot outputs. Looker’s conversational analytics is grounded in LookML. Snowflake’s Cortex Analyst uses semantic views and verified queries, and can evaluate generated SQL against known-good answers.

A 2026 paired benchmark similarly found that adding a small hand-authored semantic document improved analytics accuracy by 17–23 percentage points across three frontier models.

But an organization is not only a semantic model.

The missing context lives in decisions, exceptions, scope, and time

A semantic layer can tell an AI that revenue is net of refunds. It may not tell it that:

• the definition was changed on June 1;
• historical reports before June should not be restated;
• Enterprise contracts use a different recognition rule;
• the board dashboard still uses the previous calculation;
• Finance owns the final interpretation;
• a migration is incomplete;
• two teams currently disagree;
• the question has never actually been resolved.

“Is this dashboard authoritative?” similarly requires knowing ownership, supersession, upstream validity, and scope. This is why a context layer for AI is useful—if defined precisely.

Retrieval is not authority

RAG is good at finding relevant evidence.

That is valuable. An AI agent should absolutely be able to retrieve metric docs, dashboard descriptions, analyses, tickets, and decisions.

But relevance and authority are different properties.

Imagine the agent retrieves four highly relevant sources for “active user”:

1. a 30-day definition from last year;
2. a 28-day definition approved this quarter;
3. a live dashboard still using 30 days;
4. an unresolved ticket asking whether historical reports should change.

A retrieval system has succeeded if it finds all four.

An analytics system has not succeeded if it silently picks one.

The trustworthy answer might be:

Product reporting currently uses the approved 28-day definition. Three dashboards still use the previous 30-day logic, and the historical restatement policy is unresolved.

That answer requires representing the relationship between the sources: current, superseded, conflicting, unresolved.

Those states do not live inside any one retrieved paragraph.

Permissions are part of trust

A factually correct answer can still be wrong to provide. AI analytics has to preserve row-level access, column restrictions, domain boundaries, customer isolation, and role-specific permissions. The authoritative Finance answer may rely on evidence a Product user cannot access.

“Trusted” therefore means at least: correct, scoped, current, traceable, and authorized.

Freshness is not just a data-refresh timestamp

Analytics teams know how to think about data freshness: when did the table last update?

AI introduces another freshness problem:

When did the organization’s understanding last update?

Anthropic’s internal analytics system is a useful example. The company reports roughly 95% aggregate analytics accuracy with Claude, but says that without active maintenance of its Skills documentation, offline accuracy fell from roughly 95% to 65% over one month.

The context around how to use the data became stale. Anthropic eventually treated maintenance as an engineering problem, reporting that roughly 90% of data-model pull requests include a Skill update in the same change.

When an AI agent depends on definitions, routing guidance, and known pitfalls, they are production dependencies—not merely onboarding material.

What a trustworthy analytics answer actually requires

A reliable AI analytics system needs more than a strong model and database access.

At minimum, it needs several layers working together.

Governed data and explicit semantics

Clean models, tested transformations, reliable sources, and defined metrics, relationships, grain, synonyms, and calculation logic.

Source authority

The system should know which sources are canonical, fallback, deprecated, or scope-specific.

Temporal and scoped validity

Definitions and decisions need effective periods and boundaries: product, geography, business unit, customer type, reporting purpose, or other relevant scope.

Decisions and exceptions

The agent needs access to what the organization decided, what changed, why it changed, and which exceptions still apply.

Conflict and uncertainty

When sources disagree, the system should preserve that disagreement instead of forcing a synthetic consensus.

When there is not enough evidence, it should be able to say so.

Ownership and human authority

Someone has to be able to approve, correct, reject, or leave an interpretation unresolved.

The LLM should not silently promote its own inference into organizational truth.

Provenance

The answer should be inspectable. Users need to see the metric, source, query, decision, or evidence that supports it.

Permissions

The same context must be filtered through what the requesting user or agent is allowed to access.

Continuous maintenance

When models, definitions, or decisions change, the context supplied to AI must change with them.

Why more AI infrastructure can make the problem more visible

Before AI, ambiguity was often absorbed by analysts.

A stakeholder asked a vague question. The analyst knew that “customer” meant accounts for this team, that the old dashboard was deprecated, that Finance wanted the adjusted metric, and that the June anomaly should be excluded.

AI removes the analyst from the middle of routine questions—and exposes how much invisible context the analyst was carrying. The infrastructure has made information accessible; it has not necessarily made organizational understanding explicit.

The next analytics layer is maintained context

“Context layer” is increasingly used across enterprise AI, with competing definitions. For analytics, a useful definition is narrower:

A maintained context layer tells an AI not only what information exists, but what the organization currently allows it to rely on for a particular question—and why.

That includes semantics, but also state:

• current;
• superseded;
• conflicting;
• unresolved;
• scoped;
• time-bound;
• owned;
• evidence-backed.

Where Propperly fits

This is the problem Propperly is exploring from the analytics side.

Propperly is not meant to replace the warehouse, semantic layer, catalog, or agent platform. The hypothesis is that a missing object in the stack is the reviewed current state of analytical understanding: the definitions and decisions in force, what they replaced, where evidence conflicts, what remains unresolved, and the provenance behind those conclusions.

The goal is not more documents. It is fewer things the agent has to guess.

That hypothesis still has to be tested in real organizations. But major analytics platforms, benchmarks, and production teams increasingly point in the same direction: better models, retrieval, and semantics help; reliable AI analytics also depends on maintaining the business context around them.

References

• Spider 2.0, ICLR 2025
• EntSQL, arXiv, June 2026
• Anthropic, “How Anthropic enables self-service data analytics with Claude,” June 3, 2026
• Microsoft Learn, “Use Copilot with semantic models in Power BI”
• Google Cloud, “Conversational Analytics in Looker”
• Snowflake Documentation, “Cortex Analyst evaluations”
• Atlan, “Research Shows How Enhanced Metadata Delivers 38% Better AI Accuracy for Certain Queries”
• Rumiantsau & Fokeev, “Semantic Layers for Reliable LLM-Powered Data Analytics,” arXiv, April 2026