What a pipeline objective actually commits to
A pipeline objective states two numbers agreed with the person who owns the data: how fresh the published output must be, and how complete. Both are expressed as a target over a period, in the same way a managed data service states what it will hold to — for example, that the model is no more than four hours behind source on 99% of business days. Without both numbers there is nothing to breach, and therefore nothing to alert on. For manufacturing data operations, distinguish a delayed shift report from a machine-control event before assigning freshness targets or deciding what an incident response may change.
The framing is borrowed directly from service reliability practice. Google's SRE Workbook defines a service level indicator as a ratio of good events to total events, a service level objective as a target for that ratio, and the error budget as its inverse — a 99.9% target over three million requests permits three thousand failures. The same arithmetic works for data: if the freshness target is 99% of business days, roughly two late days a quarter sit inside budget instead of counting as incidents.
That inversion is the useful part. It converts "the pipeline should not be late", which is unfalsifiable and therefore unmanageable, into a budget that is either spent or not. A team that has spent its budget stops shipping changes to that pipeline and fixes it instead. A team with budget remaining can absorb a late day without an escalation.
The workbook also warns against a target of 100%, and the warning transfers exactly: a pipeline that is never permitted to be late cannot be changed, because every change is a risk taken against a budget of zero.
- Freshness: the maximum age of the published output, measured at the reader rather than at the last job
- Completeness: the share of expected records present, measured against a source the owner accepts
- The period and the target together — a number without a period is not an objective
Why a successful job run is not a reliability signal
A job exits zero when its code completed, not when its output is correct. An extract that returned no rows, a merge that matched nothing, a transformation that silently dropped a join — all of these succeed. This is why pipelines monitored on task status produce the characteristic failure where the business discovers the problem while the orchestrator dashboard is green.
Orchestrators are not at fault here; they do what they claim. Airflow's job is to declare dependencies, retry a failed task and let you rerun a past window — its documentation is explicit that retries are configured per operator or through the DAG's default arguments, and that a backfill runs copies of the DAG across a previous interval. None of that is a statement about whether the data is right. The orchestrator answers whether it ran; something else has to answer whether it is correct.
The practical consequence is that every pipeline needs at least one assertion about its output, not only about its execution. Most teams already have the machinery: dbt ships generic tests for uniqueness, nullability, accepted values and referential integrity, plus singular tests for anything expressible as a query that should return no rows.
Three checks that catch most of what goes wrong
Freshness, volume and shape catch the large majority of pipeline incidents between them. Freshness asks whether data arrived inside the objective. Volume asks whether the row count is plausible against recent history. Shape asks whether the columns, types and key cardinality are what the model expects. All three are cheap, and none requires knowing anything about the business meaning of the data.
Start there rather than with a comprehensive suite, because the marginal value of the first three checks is very high and the marginal value of the fortieth is usually close to zero. A suite nobody maintains degrades into noise, and a noisy suite is worse than none: it trains the responder to ignore the channel.
Business-rule checks come second, and they come from the named data owner rather than from the engineer. An engineer can assert that an order identifier is unique. Only the owner can say whether an order with a negative line total is a defect or a valid return.
- Freshness: the maximum arrival timestamp is inside the objective, checked on the published table
- Volume: row count within a plausible band of recent comparable periods, not against a fixed threshold
- Shape: expected columns present with expected types, and key cardinality unchanged
- Uniqueness on the natural key, which is the check that catches duplicate delivery
Containment protects the reader before you fix the cause
When a check fails, the first action is not to fix the pipeline. It is to decide what the reader sees in the meantime. Holding the publish so the last known-good version stays served is almost always better than publishing data that failed its own test, because a number known to be six hours old is usable and a number quietly wrong is not.
This is the highest-value habit in pipeline operations and the one most often missing. Without it, a failed quality check produces a warning in a log while the bad data flows through to the dashboard anyway — which means the test documented the incident instead of preventing it.
Containment needs the pipeline to be structured for it: the validated model and the published model have to be separable, so that publishing is a distinct action that can be withheld. Where transformation writes directly into the table the reader queries, there is no point at which to intervene.
The second containment decision is who is told. An alert that lands in a shared channel is not routed, it is broadcast. Name the responder per data product, and make the alert say what breached, which downstream models are affected, and what the reader is currently seeing.
Why quarantining beats dropping invalid rows
Invalid records should be written to a quarantine table with the reason they failed, not filtered out of the load. A dropped row is unrecoverable information: nobody can later establish whether the source sent ten thousand records or nine thousand nine hundred, and the discrepancy surfaces weeks later as an unexplained gap that costs days to investigate.
Quarantining also turns quality from a binary into a measurement. A quarantine table with a failure reason per row tells you which rule fires most often, which source produces the most defects, and whether the trend is improving — none of which is visible when the same rows are silently discarded.
dbt supports this directly through its store-failures configuration, which writes the rows returned by a failing test into an audit schema instead of reporting only a count. That turns a red test into a queryable list of exactly which records are wrong, which is the difference between a five-minute triage and an afternoon.
The one discipline quarantining requires is that somebody drains it. A quarantine table that only grows is a landfill, and the fact that nothing is ever removed from it is itself worth alerting on.
Make replay safe before you need it
Recovery depends on one property: rerunning the same window twice must produce the same result as running it once. If the load is idempotent on a natural key, a responder can rerun freely under pressure. If it is not, every rerun becomes a judgement call about duplication made at the worst possible moment, and the usual outcome is that nobody reruns anything and the gap is left in place.
Idempotency is a design property of the load, not an operational one. It comes from merging on a stable natural key with a version or sequence, from deleting and reinserting a bounded partition, or from writing a fresh partition and swapping it. It does not come from careful operators.
Backfill scope is the other half. A backfill that reprocesses everything is not a recovery procedure, it is an outage under a different name: it competes for the compute the live pipeline needs and takes long enough that the incident outlives it. Bound the window, run it in slices, and make each slice independently restartable.
Finally, a recovery that has never been run is a plan instead of a capability. Rehearse it against a non-production target on a quiet day, and time it. The number you want before an incident is how long a one-day backfill takes, and almost nobody has it when they need it.
- Idempotent on a natural key plus a version, never on arrival time
- Backfill in bounded slices, each restartable on its own
- Reconcile against the source after recovery instead of assuming the rerun worked
- Rehearse once and record the elapsed time, so the incident estimate is measured rather than guessed
Which checks should block a publish, and which should only warn
A check should fail the run when publishing would cause a wrong decision, and warn when the data is degraded but still usable. Getting this wrong in either direction is costly: too many blocking checks and the pipeline is halted nightly by non-problems, too few and the suite becomes a log of things nobody acted on. dbt expresses the distinction directly through test severity and warn or error thresholds.
The workable rule is to set severity from the consequence rather than from how important the rule feels in the abstract. A duplicate primary key breaks every aggregate built on the table, so it blocks. A null in an optional descriptive attribute degrades a filter, so it warns. The same rule can carry different severities on different models, because the consequence differs.
| Dimension | Severity | Action on failure | Who answers |
|---|---|---|---|
| Freshness beyond objective | Error | Hold the publish; the last good version stays served. | Pipeline owner, on call. |
| Duplicate natural key | Error | Hold the publish and quarantine the duplicate set. | Pipeline owner. |
| Volume outside plausible band | Error on collapse, warn on drift | Hold when the count collapses toward zero; warn on gradual drift. | Pipeline owner, with the source owner. |
| Required column missing or retyped | Error | Fail the build before deployment rather than at runtime. | Producing team, through the contract. |
| Referential integrity break | Error | Quarantine the orphans with a reason; publish the remainder only if the model tolerates it. | Data owner sets the tolerance. |
| Business rule violated | Set by the data owner | Quarantine and report; the owner decides whether it blocks. | Data owner. |
| Null in an optional attribute | Warn | Publish, track the rate, and alert on a trend instead of an instance. | Pipeline owner. |
| Late record for a closed period | Warn | Apply the stated late-arrival policy; never silently restate a published figure. | Data owner. |
Lineage is what makes an incident bounded
When a check fails, the immediate question is which reports, models and downstream products are affected. Without lineage the honest answer is that nobody knows, so either everything is treated as suspect or nothing is — and both are wrong. Lineage converts an open-ended incident into a bounded list of affected consumers who can be told what they are looking at.
The capability is increasingly available without bespoke work. OpenLineage defines an open specification built on a run, job and dataset model with extensible facets, and integrations that emit lineage events from Airflow, Spark, dbt, Flink and others. Unity Catalog captures lineage automatically across data and AI assets on Databricks. Microsoft Purview assembles the same picture across a wider estate and ties it to governance domains and data products.
What no tool supplies is the consumer list beyond the platform boundary: the spreadsheet that reads an export, the team that receives a file on a schedule. That part is a register somebody maintains by hand, and it is usually the part that makes an incident communication late.
A worked example: the source feed that arrives late
A daily finance feed that normally lands at 02:00 has not arrived by 05:00, and the reporting model publishes at 06:00. The correct sequence is: the freshness check fires at 03:00 against the objective, containment holds the publish, the named responder is paged, affected consumers are identified from lineage and told, and only then does anyone start investigating the source.
Compare that with the common sequence. The orchestrator reports no failure, because the load task never ran — it was waiting on a sensor, or it ran against an empty directory and succeeded. The model publishes on schedule with yesterday's data carried forward, or with a partial day. A finance analyst notices at 09:40 that a total looks wrong, and the first half hour of the investigation is spent establishing whether the number is wrong at all.
The difference between the two sequences is not tooling. It is that the first has an objective to breach, a publish step that can be withheld, a responder with a name and a list of who to tell. Each of those is a decision taken before the incident, and none of them can be taken during it.
What this guide does not settle
This is about the reliability of a pipeline as a system: whether it delivers what it promised, detects when it has not, and recovers safely. It is not a data quality framework — quality and observability answer different questions — and it stops deliberately short of how to define correctness. Deciding that a record is wrong requires business knowledge that no engineering practice supplies on its own.
Two further limits are worth stating. It assumes a pipeline with a single accountable owner; where ownership is genuinely shared across teams, the containment and paging decisions above need an escalation path this guide does not describe. And it assumes a cooperative source — a third-party feed that arrives late without notice is a commercial problem before it is an engineering one, and an untrusted boundary is the honest way to describe it, and no amount of instrumentation converts it into one you can fix.
Count the publication opportunities the reader depends on.
Retries are execution events; a consumer's expected publication is a service opportunity. In this hypothetical daily finance product, several runs can support one publication deadline. A reliability measure becomes misleading if successful reruns enlarge the denominator while the original missed publication disappears from the record.
Scope and assumptions: Agree the calendar, consumer cutoff, acceptable data state and observation period before measuring. These are illustrative measurement rules, not a Quorlytic SLA or an observed reliability result. A good-window ratio complements incident records and recovery measures; it does not replace them.
| Dimension | Counting rule | Evidence that makes the result reproducible |
|---|---|---|
| Eligible window | Count one expected publication for each agreed business-calendar window. Define holidays, timezones and any pre-agreed exclusions explicitly. | Keep the calendar and product scope version with the measurement. Do not remove a missed day retrospectively because the source was unavailable. |
| Good outcome | At the consumer cutoff, evaluate the agreed freshness, completeness and critical correctness conditions together. A green job is not the acceptance criterion. | Retain the version actually visible to the reader, its source cutoff and the acceptance results. Record degraded states according to the agreed rule rather than assuming last-known-good data is always fresh enough. |
| Retries | Associate reruns with the publication opportunity they were trying to satisfy. They do not create additional expected publications merely because the scheduler ran again. | Link each attempt to the same window identifier. Preserve the original outcome at the cutoff alongside the later successful recovery. |
| Partial scope | Define the product boundary, such as required regions or ledgers. A complete total for one region cannot make a multi-region product complete. | Check coverage against the agreed expected inputs and record missing partitions. Separate a product-wide measure from component-level diagnostics. |
| Changed rule | Version a changed cutoff or acceptance condition prospectively. If historical results are recalculated, retain the previous rule and explain the difference. | Keep both measurement definitions and the reason for change. Avoid improving a reported score merely by changing its denominator or exclusions after an incident. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: a scheduled publication misses its consumer cutoff, then succeeds after three retries. Record one missed publication opportunity and its later restoration, not three failed products followed by a new good product. Across twenty eligible windows, one missed window remains one breach even if many later attempts succeed. This describes a counting method, not a service commitment.
- Google's SRE guidance defines a common indicator as good events divided by total events. For this worksheet, the chosen event is the consumer publication opportunity; that mapping must be agreed for the specific data product.
- Report restoration time and unresolved data exceptions alongside the window outcome. A corrected historical dataset can be valuable without erasing what the reader experienced at the original deadline.
Carry the measurement definition and its exceptions into the managed data operating agreement before discussing monitoring responsibilities or service objectives.
Official references for this example:
Questions and trade-offs
Question
How many tests should a pipeline have?
Fewer than most teams end up with, and chosen by consequence rather than by coverage. Freshness, volume, shape and uniqueness on the natural key catch the large majority of incidents and cost almost nothing to maintain. Business rules are added by the data owner one at a time, each with a stated severity. A suite that produces alerts nobody acts on is worse than a small suite that is always believed.
Question
Should a failed quality check block the publish?
It depends on whether publishing would cause a wrong decision. Where it would — a duplicate key, a collapsed row count, data outside the freshness objective — hold the publish and keep serving the last good version, because stale and known beats fresh and wrong. Where the data is degraded but usable, publish and track the rate. The decision belongs to the data owner and should be recorded per check rather than assumed.
