Rescuing a 2-year-old dbt project

By Luca De Angelis · 8 min read

The inheritance

A B2B SaaS platform called us with a familiar brief: their dbt project was two years old, the two people who built it had left, and the nightly run now failed roughly one morning in five. Nobody wanted to touch it. Every dashboard question started with "well, it depends which table you trust."

This is the most common shape of analytics engineering consulting work we get at Satyadata. Not greenfield builds - rescues. And the first rule of a rescue is counterintuitive: do not fix anything yet.

Audit first: before changing a single model

The temptation on day one is to open the ugliest model, wince, and start refactoring. Resist it. A dbt project that has survived two years of production is a fossil record of decisions, and you need to read the record before you rearrange the bones.

Our audit took about a week and touched nothing. We just read, ran, and wrote things down.

What we looked at

• The README vs. reality. The README described a project structure that stopped being true around month six. That gap is diagnostic: it tells you when the team switched from building to firefighting.
• Naming conventions. We found three generations of conventions coexisting: stg_, base_, and a folder charmingly named tmp_final_v2. Each generation maps to an era - and usually to a person.
• Dev/prod separation. Analysts were developing against production schemas. One dbt run from a laptop could - and once did - overwrite the tables the CEO's dashboard read from.
• CI, or the absence of it. PRs were merged on visual inspection. No compile check, no tests, no dbt build on changed models. The 1-in-5 failure rate suddenly made sense.
• Materializations. Nearly every model was materialized as a table - barely a view or an incremental model in the project. The nightly run rebuilt every table from scratch and took over three hours, when less than 5% of the underlying data had actually changed since the previous run. The warehouse bill reflected it.
• Hardcoded IDs. We counted 37 hardcoded customer, plan, and country IDs sprinkled through case statements. Each one was a business rule nobody had written down anywhere else.

The ugliest model has an author, and the author has answers

Here is a habit I recommend to anyone inheriting a codebase: run git log on the ugliest model you can find, and make its most frequent author your onboarding buddy. Not to assign blame - the opposite. That person was in the room when the ugly decision was made, and the ugliness almost always encodes a real constraint: a source system that lied, a finance rule with eleven exceptions, a migration that half-happened.

In this project, the ugliest model was an 800-line revenue mart with nested CTEs seven levels deep. Its author was still at the company, in a different team. One coffee with her explained more than the entire audit week: the nesting existed because billing had changed providers mid-year and both systems had to reconcile to the cent.

Most of what gets labelled "tech debt" was never a skill problem. It was a deadline decision made by someone who knew better, under pressure, without time to write down why.

Once you see it that way, the rescue stops being a cleanup of other people's mistakes and becomes something more useful: turning undocumented decisions into documented ones.

The hardening checklist

With the audit written up, we spent roughly six weeks hardening the project. Not a rewrite - a rescue. About 80% of the models survived with modest changes.

1. Enforce a layered structure

We reorganized into the boring, standard three layers: staging (one model per source table, renaming and typing only), intermediate (business logic, joins), and marts (what BI tools and stakeholders touch). Boring is the point. Anyone who has seen a dbt project before can now navigate this one.

2. Declare sources, test freshness

Half the "pipeline failures" were actually upstream data arriving late. Declaring every input in sources.yml with freshness checks turned mystery failures into named, alarmed, upstream problems:

sources:
  - name: billing
    schema: raw_billing
    config:
      freshness:
        warn_after: {count: 6, period: hour}
        error_after: {count: 24, period: hour}
      loaded_at_field: _loaded_at
    tables:
      - name: invoices
      - name: subscriptions

3. Test the contracts, not everything

We added two standard open-source packages - dbt_utils for the common data quality checks, and Elementary for anomaly detection and alerting - and put tests where they earn their keep: primary keys unique and not null on every staging model, accepted values on status columns, and anomaly detection on row counts for the five business-critical marts. Around 300 tests total - enough to catch real breakage, few enough that a red test still means something.

4. CI on every pull request

Every PR now compiles the project, builds the changed models plus their first-order children into a disposable schema, and runs their tests. Merge is blocked on green. This did more for the team's confidence than any other single change: within a month they went from batching changes into nervous weekly releases to merging small changes most days.

5. Alerts where the team lives

Elementary pipes run results and anomalies into a Slack channel, with an owner tagged per mart. A failure at 6 a.m. is now a Slack message to one accountable human, not a surprise in a stakeholder meeting at 10.

6. Style enforced by robots, not reviews

SQLFluff as a pre-commit hook ended the era of code reviews about comma placement. Humans review logic; the linter reviews style. Nobody has argued about leading commas since, which may be the project's most measurable quality-of-life gain.

Making the project AI-ready

Here is the step most rescues still skip in 2026. A clean dbt project is no longer just for humans - it is context for AI coding agents, and the returns on preparing it are immediate.

• Context files at the root. We added CLAUDE.md and AGENTS.md files describing the project layout, the warehouse, how to run and test models, and the three business rules that surprise everyone (that billing migration, for instance). An agent that reads this file starts with the knowledge our onboarding buddy carried in her head.
• Conventions as Markdown, not folklore. Naming rules, layer definitions, and materialization policy live in a short conventions.md. If it is written down, both the new analyst and the coding agent follow it. If it is folklore, neither does.
• Decisions with a why. Every hardcoded ID we kept got a comment linking to a one-paragraph decision note. The next deadline decision will at least leave a trail.

The payoff compounds: the client's analysts now use AI assistants to draft staging models and tests, and the assistants are actually good at it - because the project finally tells them what "good" looks like. I've written more about this workflow in how I work as an AI-enhanced data analyst: agents don't get smarter, they get context.

Results

• Failed nightly runs went from roughly one in five to about one a month - and those now alert the right person before stakeholders notice.
• Warehouse spend on the nightly run dropped around 40% after moving the right models to incremental and view materializations.
• Time to first merged PR for a new analyst fell from about three weeks to three days.
• The revenue mart shrank from 800 lines to five readable models, each reconciling to the cent - with the "why" documented this time.

What to take away

If you are inheriting a dbt project: audit before you touch, befriend the author of the ugliest model, and treat tech debt as decisions to be documented rather than sins to be punished. Then harden the boring things - layers, sources, tests, CI, alerts, linting - and write your conventions down in Markdown so both humans and AI agents can follow them.

And once the pipes are trustworthy, point them at something worth measuring - a messy project and a messy metric layer usually travel together, which is why this engagement flowed straight into defining a KPI framework and North Star metric for the same client.

A rescued project is not one with no ugly models. It is one where every ugly model can explain itself.