What separates a contract from documentation
A contract is enforced; documentation is aspirational. That makes it a data engineering artefact instead of a governance one. The practical test is simple: if a producer removes a column tomorrow, does anything stop? If the answer is that a consumer's pipeline fails the following night, you have documentation. If a build fails before the change is merged, you have a contract. Everything else in the design follows from where that check runs.
This matters because the two look identical on paper. A schema document listing columns, types and meanings is the same artefact in both cases. The difference is entirely in whether something reads it at a moment when a person can still change their mind — and that moment is before merge, not after deployment.
It also changes who the contract is for. Unenforced documentation is written for consumers, to tell them what to expect. An enforced contract is written for the producer, because it is the producer whose change gets blocked. That reframing is usually what makes the idea land with the producing team: the contract is not paperwork imposed on them, it is the thing that stops them causing an incident they will be paged for.
What belongs in a contract, and what does not
A contract covers four things: the structure (column names, types, nullability, keys), the semantics (what each field means and its unit), the delivery promise (cadence, expected lateness, how a failure is communicated), and the change rules (which compatibility rule applies and how long a deprecation window lasts). It does not cover implementation, storage format or how the producer computes the value.
The semantic half is the part most often omitted and the part that causes the most expensive confusion. A column typed as a decimal tells a consumer nothing about whether the amount is gross or net, in which currency, inclusive of tax, or in minor units. Schema enforcement will happily pass a field that changed from pence to pounds, and every downstream figure will be wrong by a factor of a hundred with no test failing anywhere.
The delivery promise is the second omission. A consumer building on a dataset needs to know whether it lands daily at 03:00 or whenever an upstream job happens to finish, and what happens when it does not arrive. Without that, every consumer invents its own assumption, and those assumptions are discovered during an incident.
Keeping implementation out is equally important. A contract that specifies the producer's internal table structure or processing engine cannot be honoured through a refactor, so it gets abandoned the first time the producer needs to change something legitimate. A contract constrains the interface, not the work behind it.
- Structure: names, types, nullability, the natural key, and cardinality expectations
- Semantics: meaning, unit, currency, time zone, and whether the value is gross or net
- Delivery: cadence, expected lateness, and how a failure is communicated and to whom
- Change: the compatibility rule, the deprecation window, and who is notified
Choosing a compatibility rule decides who upgrades first
A compatibility rule is a statement about which side of the interface can move independently. Confluent's schema registry names them precisely: BACKWARD permits a consumer on the new schema to read data written with the old one and requires consumers to be upgraded first; FORWARD is the reverse and requires producers first; FULL permits only optional additions and removals and lets either side move independently.
The choice is an organisational decision wearing technical clothing. BACKWARD suits an estate where the producing team is the constraint and consumers are numerous and cooperative. FORWARD suits the opposite. FULL is the most restrictive about what may change and the least demanding about coordination, which makes it the right default across a team boundary where you do not control the other side's release schedule.
The transitive variants matter more than they appear to. Non-transitive compatibility is checked only against the latest version, so a sequence of individually compatible changes can leave a consumer that has not upgraded in several versions unable to read anything. Where consumers move slowly — and outside your own team they usually do — the transitive variant is the one that matches reality. Confluent notes that BACKWARD non-transitive is the default, which means the safer behaviour is the one you have to opt into.
NONE is worth naming explicitly because estates arrive at it by accident. It disables checking entirely and requires producers and consumers to be upgraded simultaneously, or the topic to be migrated. That is a coordination cost paid on every change, and it is rarely chosen deliberately.
Where enforcement actually runs
Enforcement belongs wherever the change is made, which in practice means three places: a schema registry at a streaming boundary, a build-time contract in the transformation layer, and a test in the consumer's own pipeline. The first two block the change before it ships. The third only detects it afterwards, which is why it is a supplement instead of a substitute.
In the transformation layer, dbt model contracts do this directly. With a contract enforced, dbt requires explicit column names and data types, runs a preflight check that the model's query returns columns matching the contract, and includes declared constraints in the DDL it submits. A build fails when a contracted column is removed, when a column's data type changes, when constraints are removed or modified, or when a contracted model is deleted, renamed or disabled.
Constraint support varies by platform, and that variation is worth knowing before relying on it. dbt's documentation is explicit that platforms differ — Postgres enforces the full set, while Snowflake enforces only not-null. A contract whose constraints are not actually enforced by the warehouse is back to being documentation, so the build-time check is doing the real work in those cases.
At a streaming boundary the registry serves the same function earlier, refusing an incompatible schema at registration rather than at read time. The two mechanisms are complementary: the registry governs the event as published, the model contract governs the table as served.
How to remove a field without breaking anyone
Removal is a sequence, not an event: announce the deprecation with a date, add the replacement alongside the old field, monitor who still reads the old one, notify the remaining consumers individually, and only then remove it. The window has to be long enough for a consumer's normal release cycle, which means it is set by the slowest consumer rather than by the producer's convenience.
The step teams skip is monitoring actual reads, and it is the one that makes the rest safe. A deprecation announcement tells you who replied; query logs or access telemetry tell you who is still reading. Those two lists are rarely the same, and the difference is exactly the set of consumers who will be surprised.
Versioning is the alternative where the change is too large to be additive. Publishing a version 2 alongside version 1, with both served for an agreed period, converts a breaking change into a migration consumers can schedule. It costs the producer a period of dual maintenance, which is real, and it is almost always cheaper than the coordinated cutover it replaces.
A deprecation window with no end date is not a deprecation. If no removal date is set, the old field is permanent, and the estate accumulates fields nobody dares delete — which is how a table reaches a hundred and forty columns of which eleven are read.
| Dimension | Breaking? | Safe handling |
|---|---|---|
| Add an optional column | No | Ship it. Consumers that do not select it are unaffected. |
| Add a required column | For producers, yes | Add as optional, backfill, then tighten once every writer supplies it. |
| Remove a column | Yes | Deprecate with a dated window, monitor reads, notify remaining consumers, then remove. |
| Rename a column | Yes | Treated as add plus remove. Never rename in place. |
| Widen a type | Usually not | Ship it, but check consumer-side casts that assume the narrower range. |
| Narrow a type | Yes | New column or new version; existing values may no longer fit. |
| Change a unit or currency | Yes, and silently | New column with the unit in its name. Never reuse the old column. |
| Change the grain of a row | Yes, severely | A new version of the dataset, with both served during migration. |
| Add a value to an enumeration | Depends on the consumer | Announce it; consumers with exhaustive handling need a default branch first. |
| Change delivery cadence | Yes, operationally | Announce with a window; downstream schedules and freshness checks depend on it. |
Where contracts are worth the overhead, and where they are not
Contracts earn their cost at a boundary between teams who release on different schedules, and where a break would be expensive. Inside a single team owning both sides, the overhead usually exceeds the benefit: the same people review both changes, and a test is sufficient. Applying contracts uniformly across every internal model is the fastest way to make the practice unpopular.
The useful triage is by blast radius and by distance. A dataset read by three teams, by a regulatory report or by a customer-facing product needs a contract. An intermediate model read only by the next model in the same project, written by the same person, does not — it needs a test, and its interface changes in the same pull request as its consumer.
There is a second condition that is easy to miss: a contract needs a producer who can be held to it. Where the producer is a third-party SaaS vendor, you do not have a contract, you have a hope and a monitoring requirement. Treat that boundary as untrusted, validate on arrival, and keep the quarantine path, because no compatibility rule you write governs their release notes.
The organisational precondition
A contract requires a named producer who is accountable for the dataset. Where the data simply arrives from a system with no owner — which is common for older operational databases — there is nobody to sign the contract and nobody whose deployment can be blocked. Establishing that ownership is the prerequisite, and it is a governance task instead of an engineering one.
This is where data contracts and the governance operating model meet in practice. The catalogue model most vendors implement makes the same assumption: Microsoft Purview's framing assigns data owners who register assets and hold quality standards, and stewards responsible for quality and lineage, with a central office setting policy. The contract is the mechanism through which that ownership becomes visible at a technical boundary.
Where ownership genuinely cannot be established, be honest about what is possible. You can validate on arrival, quarantine what fails and alert on drift. You cannot prevent the change. Calling that a contract overstates the control you have, and the overstatement surfaces at the worst moment.
What this guide does not cover
It does not specify a format. A contract can be a schema in a registry, a YAML specification in a repository, a model contract in the transformation layer or a table definition with constraints, and the right answer depends on where the boundary sits. The argument here is about what a contract must contain and where it must be enforced, not about which file it lives in.
It also stops at the technical boundary. A contract can guarantee that a field is present, typed correctly and delivered on time. It cannot guarantee that the value is right — a correctly shaped, punctually delivered number can still be computed from the wrong source. That is a quality and observability question, and it needs its own controls alongside the contract rather than inside it.
Questions and trade-offs
Question
Do we need contracts on every dataset?
No, and trying to is the most reliable way to make the practice fail. Contracts earn their overhead at a boundary between teams releasing on different schedules, where a break would be expensive — a dataset read by several teams, a regulatory report, a customer-facing product. An intermediate model read only by the next model in the same project needs a test, because its interface changes in the same review as its consumer.
Question
What do we do about sources we do not control?
Treat the boundary as untrusted instead of pretending it is a contract. Validate structure and semantics on arrival, quarantine what fails with a reason, monitor for drift in shape and distribution, and keep a replayable raw landing so a late-discovered change can be reprocessed. You cannot block a third party's release, so the control moves from prevention to detection and recovery — and it should be described that way honestly.
