AI data analyst: why context beats a bigger model

By Luca De Angelis · 8 min read

Every analysis I run these days starts the same way. I type "start ticket 4711" and an agent goes to work on my machine. Not a chatbot in a browser tab - an AI data analyst that lives inside my workspace, with my repos, my pipelines and two years of past analyses at its fingertips.

Here is the uncomfortable part: the agent running today uses roughly the same class of model I had access to a year ago. Back then, it got analytical tasks right maybe one time in four. Today it lands around 94%. The model did not get four times smarter.

The context did.

What "start ticket" actually triggers

When I kick off a ticket, the agent does not start writing SQL. That is the mistake almost every "AI analyst" product makes - jumping straight from a vague question to a confident query. Mine goes through a sequence that looks a lot like what a good senior analyst does on their first week at a new company, compressed into a few minutes.

Step 1: Read the workspace

The agent reads the local workspace: the dbt project, the pipeline repos, the orchestration configs, and - this is the underrated part - a folder of past analyses. Every ticket I have ever closed left behind a markdown file with the question, the findings and the queries that produced them. That folder is the institutional memory most data teams never write down.

Step 2: Narrow down where the logic lives

Say the ticket is "conversion rate dropped 2 points in the Netherlands last week". The agent's first job is not to answer it - it is to find where conversion is actually defined. Which model computes it, which sources feed that model, which past tickets touched the same metric, whether anyone shipped a change to that pipeline recently. In a mature warehouse, "where does this number come from?" is 60% of the work. I wrote about why tracing a number back to its source matters in rescuing a 2-year-old dbt project - the same audit instinct applies here, just automated.

Step 3: Write the ticket file - then stop

The agent writes everything into a ticket.md: the goal as it understood it, what it found in the codebase, and a ranked list of hypotheses. Something like this:

# Ticket 4711 - NL conversion drop, week 27

## Goal
Explain the ~2pp drop in NL checkout conversion (wk 26 to wk 27).

## Findings
- conversion defined in marts/fct_checkout_funnel.sql
- NL sessions route through a separate payment provider since 2026-06-30
- past ticket 4102: similar drop in AT traced to bot traffic

## Hypotheses (ranked)
1. Payment provider migration broke a checkout step for iDEAL
2. Bot/paid-traffic mix shift inflating the denominator
3. Tracking gap after last week's consent-banner release

## Status: AWAITING REVIEW - no queries run yet

And then it stops. Deliberately. That pause is the single most important design decision in the whole system.

Because this is the human judgment step. I read the hypotheses and bring what the agent cannot know: that the payments team mentioned a migration in standup, that hypothesis 3 came up last quarter and was a dead end, that the stakeholder actually cares about revenue, not conversion. I reorder, cut, add. Thirty seconds of judgment that saves three hours of confident nonsense.

Step 4: One query per hypothesis, every result back into the ticket

Only now does the agent generate SQL - one focused query per approved hypothesis, run against the warehouse, with every result appended to the ticket file. Hypothesis confirmed, rejected, or "inconclusive, here is the follow-up query". By the end, ticket.md is not a chat transcript that evaporates when I close the tab. It is a self-contained investigation record: question, evidence, queries, conclusion. It gets committed. It becomes context for ticket 4712.

Agents don't get smarter. They get context. Everything that looks like intelligence in this workflow is really curation: what the agent is allowed to read, when it must stop, and what it is forced to write down.

Where the 26% to 94% jump really came from

When I first pointed an agent at a client warehouse, I measured how often it produced a correct, decision-grade answer to a realistic ticket. It was around 26%. Not because the model was weak - because it was guessing. Guessing which table was the source of truth, guessing what "active user" meant, guessing that a column called revenue contained revenue. (It contained gross bookings. Of course it did.)

Over the following months, accuracy on the same class of tasks climbed to roughly 94%. Here is what changed, in order of impact:

• Past analyses as memory. The folder of closed tickets. The agent stopped rediscovering known dead ends and started citing precedent.
• The forced stop. Splitting the workflow into "investigate, then wait" killed the failure mode where one wrong early assumption poisons everything downstream.
• Metric definitions as code. When "conversion" resolves to one dbt model instead of tribal knowledge, the agent cannot pick the wrong one.
• The ticket file itself. Writing structured state to disk means every step reasons over accumulated evidence, not a decaying chat history.
• Model upgrades. Real, but honestly worth maybe 10 of those 74 points. The rest was structure.

That ordering is the whole thesis. Teams waiting for the next model release to make their AI data analyst reliable are waiting for the wrong train.

Data is not software

Coding agents got good fast, and everyone assumed analytics agents would follow one release later. They did not, and the reason is structural: data is not software.

Code carries its own ground truth. If an agent writes a function that does not compile or fails the tests, it knows immediately and self-corrects. That tight feedback loop is why coding agents work.

A SQL query that runs and returns a plausible-looking number has passed no test at all. Wrong join, silently duplicated rows, a filter that excludes half of Germany - the query "works". There is no compiler for meaning. The feedback loop that coding agents get for free has to be built for data work: the ticket structure, the human checkpoint, the inspectable queries, the accumulated memory. That is also why trust has to be engineered rather than assumed - a topic big enough that I gave it its own article on what makes an AI analytics agent trustworthy.

An AI data analyst inside a client team

We have set this workflow up for a handful of Satyadata clients now - a marketplace, a B2B SaaS platform - and the pattern repeats. The build is maybe 20% agent plumbing and 80% context engineering: consolidating metric definitions, structuring the repo so logic is findable, seeding the memory folder with the ten analyses everyone keeps redoing.

The payoff shows up in analyst time. At the SaaS client, routine investigative tickets - "why did metric X move?" - went from a day or two of an analyst's week to under an hour of review time, with the agent doing the excavation and the human doing the judgment. The analysts did not get replaced. They got promoted, in practice, from archaeologists in search of logic and data sources to editors.

And a second-order effect I did not expect: because every investigation now ends in a committed markdown file, the team accidentally built the analytical knowledge base they had been "meaning to write" for three years.

Takeaway

If your AI data analyst experiments are stuck at "impressive demo, useless in production", the fix is almost never a bigger model. Give the agent a workspace worth reading, force it to write down what it thinks before it acts, and put one human judgment step exactly where it hurts to skip it.

Structure is the upgrade.