The Transformation Layer That Changed How Data Teams Work
For years, the analytics engineering function sat in an uncomfortable middle ground. Data engineers built pipelines to land raw data into warehouses, and analysts wrote SQL to query it — but the code that transformed that raw data into something useful was often a scattered collection of one-off scripts, undocumented stored procedures, and queries nobody dared to touch. There was no version control, no testing, and no shared understanding of what a "customer" or an "order" actually meant inside the database. dbt — the data build tool — was built to fix exactly that problem, and in doing so it has become the de facto transformation layer in what the industry now calls the modern data stack.
At its core, dbt is a command-line tool that allows analysts and engineers to write transformations as plain SQL SELECT statements, then layers on top of that a software engineering workflow: version control, modular code, automated testing, and documentation that lives next to the code that generates it. The result is that what was once tribal knowledge locked in a single analyst's head becomes a reviewable, auditable, testable codebase that a whole team can own and evolve.
Why dbt Won the Modern Data Stack
The modern data stack coalesced around a set of architectural decisions that made dbt's approach uniquely well-suited to the moment. Cloud data warehouses like Snowflake, BigQuery, and Databricks made compute cheap and elastic, removing the old constraint that transformations had to be done sparingly because warehouse compute was expensive. That shift meant it was suddenly practical to run complex, multi-step SQL transformations inside the warehouse itself rather than in a separate processing layer — which is precisely the model dbt is built on.
Before dbt, the dominant approach to transformation was ETL: extract, transform, load, where transformations happened before data landed in the warehouse, often inside proprietary tools that were opaque to the analysts who needed to understand the data. dbt flipped this to ELT — extract, load, transform — pushing transformations into the warehouse where SQL analysts already lived. This was not merely a technical preference; it was a democratisation. Suddenly the people who understood the business questions most deeply could also own the code that answered them, without needing to know Python, Spark, or Java.
dbt also arrived at precisely the moment teams were beginning to take data quality seriously. Organisations were burning themselves on dashboards that showed contradictory numbers, on reports built on subtly broken joins, on metrics that changed meaning silently after a schema update upstream. dbt gave teams a practical on-ramp to automated testing without demanding they first rebuild their entire infrastructure.
The Medallion Architecture and Why It Matters
One of the most consequential patterns that dbt projects tend to adopt is the medallion architecture, a layered approach to organising models that separates raw data from business logic in a principled way. The architecture is typically described in three layers: bronze (or staging), silver (or intermediate), and gold (or mart).
The staging layer sits closest to the raw source data. Models here do the minimum necessary work — renaming columns to a consistent convention, casting types, and light cleaning — but they do not join tables or apply business logic. The discipline of keeping staging models thin is important: it means there is always a layer of the project that faithfully mirrors what came out of the source system, and any analyst can inspect it without navigating layers of transformation to understand what they are looking at.
The intermediate layer is where business logic begins to accumulate. Joins happen here, derived fields are calculated, and the messy reality of source data starts to be shaped into concepts that reflect how the business actually operates. Keeping this work in its own layer prevents the gold layer from becoming an unmanageable tangle of logic that is simultaneously trying to clean data, apply rules, and serve end users.
The mart layer — gold — is what dashboards, reports, and downstream consumers actually query. These models are wide, denormalised, and built to answer specific business questions. Because the upstream layers have already handled cleaning and logic, mart models tend to be readable and relatively simple, which matters when an analyst who did not write the model needs to understand or modify it later.
This separation of concerns is one of the things that makes large dbt projects maintainable over time. Without it, transformation code tends toward a kind of entropy where every model depends on every other model in unpredictable ways, and a change to a single upstream table can break downstream models in ways that are difficult to trace.
Generic Tests and the Culture of Data Quality
dbt ships with a built-in testing framework, and its generic tests are one of its most practically powerful features. Out of the box, dbt provides four generic tests that can be applied to any column in any model with a single line of YAML: not_null, unique, accepted_values, and relationships. These cover the overwhelming majority of the data quality assertions that teams actually care about in day-to-day work.
The not_null and unique tests are self-explanatory and deceptively powerful — a primary key that is neither null nor duplicated is the foundation of any trustworthy model. The accepted_values test allows teams to assert that a categorical column only ever contains values from a defined list, which surfaces upstream data problems early rather than letting bad values propagate silently into dashboards. The relationships test is a referential integrity check: it asserts that every value in a foreign key column exists in the referenced table, catching join problems before they reach end users.
What makes these tests genuinely shift team culture is that they are declarative, version-controlled, and run automatically as part of the build process. Testing is no longer something a team resolves to do eventually; it is baked into the workflow from the first model. Teams that adopt dbt consistently report that automated testing changes how they think about data changes — a schema change upstream becomes something to test against rather than something to discover in production.
The Macro System and Reusability at Scale
As dbt projects grow, teams inevitably encounter patterns that repeat across models: the same date spine logic applied in a dozen places, the same approach to handling late-arriving events, the same surrogate key generation pattern used everywhere. Without a mechanism for reuse, teams copy and paste this logic, and the project accumulates subtle inconsistencies as copies drift apart over time.
dbt's macro system solves this problem by bringing Jinja templating into SQL. Macros are reusable snippets of SQL or logic that can be called from any model in the project, parameterised and composed just as functions would be in an application codebase. A team can write a single macro that generates a surrogate key, call it consistently across every model that needs one, and update the logic in one place when requirements change.
The macro system also powers dbt's package ecosystem. Packages like dbt-utils and dbt-expectations extend the built-in functionality with additional generic tests, utility macros, and common patterns that teams would otherwise have to build themselves. This shared infrastructure means that the solutions individual teams develop to common problems accumulate into reusable community assets rather than being rediscovered from scratch by every organisation that hits the same issue.
Software Engineering Discipline for Data Teams
Perhaps the deepest thing dbt has contributed to the analytics engineering field is a cultural shift toward treating data transformation code with the same discipline that software teams apply to application code. Version control, code review, automated testing, modular design, documentation, and continuous integration are all standard practice in software engineering, and for good reason — they are the mechanisms by which codebases remain understandable and changeable as they grow. Data teams, historically, have operated without most of these practices, and the results have been predictable: brittle pipelines, undocumented logic, and a growing fear of changing anything that currently works.
dbt does not impose these practices by fiat. It makes them natural by building them into the tool's workflow. A dbt project is a git repository. Models are files with clear inputs and outputs. Tests are declarations that run automatically. Documentation lives in the same YAML files that define the tests. The path of least resistance inside a well-structured dbt project is the path of good engineering practice, which is why it has been adopted so widely by teams that never thought of themselves as software engineers at all.
For organisations building out data capabilities, dbt represents not just a better way to write SQL, but a coherent foundation for a data engineering practice that can scale without collapsing under its own weight. The teams that get the most out of it are those that lean into its conventions fully — layering their models, testing their assumptions, and building macros for anything that repeats — and in doing so, they build something that looks increasingly like the kind of maintainable, collaborative codebase that software teams have been building for decades.