What each one is actually asking
A quality check asks whether a value satisfies a rule that someone decided in advance: this column is unique, this value is within an accepted set, this key exists in that table. An observability signal, of the kind an operated data platform reports on, asks whether what just happened resembles what usually happens: the row count, the arrival time, the null rate, the distribution of a column. The first needs a definition. The second needs history.
That difference in input is the whole distinction, and it determines everything downstream. Because a quality rule is a statement about correctness, it can justify blocking a publish: the data violates something the business asserted. Because an observability signal is a statement about similarity, it usually cannot: today's data being unusual is not the same as today's data being wrong, and quarter-end is unusual every quarter.
It also determines coverage. Quality coverage is bounded by imagination — you test what you thought of. Observability coverage is bounded by history and by what is measurable, which means it can flag a column nobody wrote a rule for, and will also flag legitimate change as though it were a defect.
What a quality rule catches that observability will not
A quality rule catches a defect that is invisible in the aggregate. If a hundred orders have a duplicated identifier in a batch of two million, the row count is normal, the arrival is on time, the null rate is unchanged, and every distribution looks exactly as it should. Nothing about the shape of the data is unusual. Only a uniqueness test finds it.
This class of defect is common and consequential, because it is precisely the kind that survives to a report. Duplicate keys inflate sums and counts while leaving distinct counts correct, which produces the characteristic situation where two numbers on the same dashboard disagree and both look plausible.
dbt's generic tests cover the common cases directly: uniqueness, nullability, accepted values and referential integrity, expressed on the model rather than in a separate system. Singular tests extend that to anything a query can express as rows that should not exist. The important property is that the assertion lives with the model it describes, so it travels when the model is changed — the same reasoning that puts transformation under orchestration.
Severity is what makes these operational rather than decorative. dbt lets a test warn or error, with thresholds, and setting that per test is how a suite stays believable: the ones that block are the ones whose failure would cause a wrong decision.
What observability catches that no rule anticipated
Observability catches the failure nobody wrote a rule for, which is most of them. A source system changing an enumeration, a feed silently halving because one upstream region stopped sending, a join beginning to drop rows after a schema change, a timestamp switching time zone — none of these violate a rule anyone thought to write, and all of them change how the data behaves.
This is the genuine argument for it. A quality suite is a record of the failures a team has already experienced or imagined; it cannot cover a mode the team has not encountered. Behavioural monitoring has no such limit, because it compares against the data's own past rather than against a specification.
The corresponding weakness is false signal. Legitimate change looks exactly like a defect: a marketing campaign, a new market launch, a pricing change and a public holiday all shift volume and distribution. An observability practice without a triage habit produces alerts that get muted, and a muted signal is worth nothing at the moment it is finally correct.
The practical mitigation is to treat observability output as a question routed to someone who knows the business context, not as an incident. A signal that says "order volume from one region is 40% below its usual band" is useful to a person who knows whether that region had a holiday. It is noise in a channel nobody owns.
Why neither substitutes for the other
They fail in complementary places, which is exactly why one does not replace the other. Quality has perfect precision on the rules it holds and zero coverage beyond them. Observability has broad coverage and imperfect precision everywhere. An estate with only quality tests misses everything unanticipated; an estate with only observability cannot block a publish, because it never has grounds to assert that the data is wrong.
The sequencing that works in practice is to start with a small quality suite on the things that would cause a wrong decision, then add behavioural monitoring across the estate for breadth, and route both to the same owner. Starting with observability alone is tempting because it requires no definitions and shows results immediately, and it leaves the estate unable to stop bad data reaching a reader.
There is one more asymmetry worth naming: quality rules are an asset that accumulates, and observability baselines are not. Every rule written down is a piece of business knowledge captured permanently, usually during an incident when someone finally states what should have been true. Those statements are worth more than the tests they become.
| Dimension | Data quality | Data observability |
|---|---|---|
| The question | Does this value satisfy a rule we stated? | Is this data behaving as it usually does? |
| Input required | A definition, decided by a person who knows the business. | History of the dataset's own behaviour. |
| Typical signals | Uniqueness, nullability, accepted values, referential integrity, business rules. | Row volume, freshness, null rate, distribution, schema drift. |
| Catches | Anticipated defects, including ones invisible in aggregate. | Unanticipated change, including modes nobody has seen before. |
| Misses | Anything nobody thought to write a rule for. | Small defects that do not move an aggregate. |
| Output | A verdict: this is a defect. | A question: this looks unusual. |
| Can it block a publish | Yes, and it should when the consequence warrants it. | Rarely — being unusual is not being wrong. |
| Dominant failure mode | Incomplete coverage; the suite reflects past incidents only. | False signal, then alert fatigue, then a muted channel. |
| Who owns it | The data owner, who states the rule. | The platform or pipeline owner, who triages the signal. |
| What it leaves behind | Accumulated business knowledge, written as assertions. | A baseline that resets as the data changes. |
Both need somewhere to send the answer
The most consistent failure in both practices is organisational rather than technical: signals with no named recipient. A quality test that fails into a build log and an observability alert that lands in a shared channel have the same effective coverage, which is none. The routing decision — who is told, and what they are expected to do — is what converts either one into a control.
Routing needs two things that estates often lack. The first is an owner per data product rather than per system, because a quality defect is a question about meaning and only the owner can answer it. The second is lineage, so the alert can say what is downstream: an alert that names the affected reports is actionable, and one that names a table is a research task.
The catalogues make the first part easier than it used to be. Microsoft Purview's model assigns data owners who register assets, manage classification and access and hold quality standards, and data stewards responsible for quality, discovery, glossary consistency and lineage — with a central office setting policy. The tool does not create the accountability — the operating model does — but it does give it somewhere to live so it stops being informal.
How this connects to AI systems
Retrieval-based AI systems inherit both problems and make the second one worse. A defect that would be visible in a dashboard as a wrong total is invisible in a generated answer, because the answer is fluent prose with a citation instead of a number with a history. The reader has no baseline against which the output looks wrong.
That changes the balance between the two practices. In a reporting estate, an experienced analyst is a genuine last line of detection — they notice when a number looks off. In a retrieval system there is no equivalent, so more of the burden falls on controls that run before the data is indexed rather than on a reader who might catch it after.
The specific quality dimensions that matter also shift. Uniqueness and referential integrity matter less to a retrieval index than currency and authority: whether the retrieved document is the current version, whether a superseded policy was removed, whether the record the reader is not entitled to see was excluded, which is a retrieval design question as much as a quality one. Those are quality rules, and they have to be stated by somebody who knows the corpus.
A starting point that does not collapse
Begin with four quality rules per critical data product and behavioural monitoring on arrival and volume across everything else. Four rules is small enough to maintain and large enough to cover the defects that cause wrong decisions. Monitoring on arrival and volume is cheap, needs no definitions, and catches the failure modes that account for a large share of incidents.
Expand from incidents rather than from a framework. When something goes wrong, the post-incident question is whether a rule could have caught it, and if so, that rule gets written and gets a severity. A suite grown this way stays believed, because every test in it exists for a reason somebody remembers.
Resist the opposite approach — adopting a comprehensive dimensional framework and attempting to instrument all of it at once. It generates a large suite quickly, most of which nobody owns, and its main lasting effect is to teach the team that a red check does not necessarily mean anything.
- Four rules per critical data product, chosen by consequence and owned by the data owner
- Arrival and volume monitoring across the estate, triaged by the pipeline owner
- Every incident ends with a decision about whether a rule should exist
- One routing destination per data product, named, with lineage attached to the alert
What this comparison does not settle
It does not recommend tools, and the choice between an observability platform, a catalogue with quality features and rules expressed in the transformation layer depends on the estate rather than on the distinction drawn here. It also does not define quality dimensions formally; several credible frameworks exist and they disagree at the edges.
One limitation is worth stating plainly: neither practice tells you whether the data is right in the sense that matters commercially. A perfectly unique, complete, fresh and normally distributed dataset can still be measuring the wrong thing. That is a modelling and definition problem, and no amount of monitoring detects it.
Check the detector with a known defect.
A configured rule is not evidence that the right failure will reach the right owner. This illustrative detector worksheet uses synthetic order data to compare a business assertion with a behavioral signal. It includes one valid change as well as defects, so that acceptance considers unnecessary alerts and missed problems together.
Scope and assumptions: Run controlled fixtures in an isolated or expressly permitted test path. Set the dataset grain, expected source coverage and clock semantics first. These cases describe desired detection behavior; they do not claim that a particular tool automatically diagnoses the cause or that any detector is infallible.
| Dimension | Expected signal and blind spot | Evidence to retain |
|---|---|---|
| Duplicate key | At a declared one-row-per-order grain, inject a repeated order identifier while keeping total volume plausible. A configured uniqueness assertion should identify the defect; a volume signal may stay normal. | Retain the fixture, declared grain, failing rule result and accountable recipient. Check that a legitimate repeated line-item key is not tested against the wrong grain. |
| Missing region | Remove one required region while another region grows, leaving the total similar. An aggregate volume monitor may miss the change; coverage checks or segmented behavior can expose it. | Record the expected region set and the affected consumer. Verify that the alert explains missing coverage rather than relying on an apparently normal total. |
| Old source data | Load an old source snapshot successfully at the expected time. An ingestion timestamp can look fresh while the business events remain stale. | Capture both clocks and the monitored timestamp field. Check the consumer's required event cutoff separately from whether the loader recently ran. |
| Wrong period | Keep keys and totals valid but assign a boundary event to the wrong business day. Generic uniqueness and null checks can pass while the business result is incorrect. | Keep the expected timezone and period assignment with the fixture. Use a business-specific assertion; a generic test inventory does not establish semantic coverage. |
| Valid demand | Introduce a legitimate promotion-driven increase with valid records. Behavioral monitoring may flag the change even though the tested business rules pass. | Record the agreed investigation and disposition. Preserve evidence that the change was explained; do not disable all volume monitoring merely to silence one understood event. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the fixture with a missing region produces no alert because the monitor only checks the total count. Add or refine the coverage signal and repeat that fixture, then rerun the valid-promotion case to examine alert behavior. The objective is evidence about detection boundaries, not a larger count of enabled tests.
- dbt supplies generic assertions such as uniqueness, non-null values, accepted values and relationships. Their usefulness depends on the declared grain and business assumptions; retain custom assertions where the question cannot be expressed by those generic rules.
- For every case, retain the input version, expected outcome, actual result, routing evidence and disposition. Recheck the fixture when the rule, model or source definition changes.
Connect detector evidence to the consumer cutoff and containment decisions in the pipeline reliability measurement guide. Detecting an anomaly and deciding what the reader should see are separate responsibilities.
Official references for this example:
Questions and trade-offs
Question
If we buy an observability platform, do we still need tests?
Yes. An observability platform compares data against its own history and is very good at surfacing what nobody anticipated, but it cannot assert that a value is wrong — only that it is unusual. Defects that do not move an aggregate, such as a small number of duplicate keys in a large batch, are invisible to it and are exactly the ones that reach a report. The two are complementary, and the tests are also where business knowledge is captured.
Question
Where should quality rules live?
As close to the model they describe as possible, so they travel with it when it changes. Rules expressed in the transformation layer are versioned with the logic, reviewed in the same pull request and run in the same pipeline, which makes them far harder to leave behind than rules held in a separate system. Where a rule has to span systems, a catalogue is the right home — but the default should be alongside the model.
