Pipeline reliability and the failures that never raise an alert

Pull-quote: “A pipeline that crashes is doing you a favour. The expensive one finishes in four minutes and writes an empty partition.”

The data pipeline failures that destroy trust are the ones that complete successfully. A job that throws an exception wakes someone up, gets a ticket, and is fixed by lunchtime. A job that reads zero files because an upstream export never landed, writes an empty partition, and reports success sits undetected until an analyst notices that last Tuesday looks quiet.

This post is for the engineer who owns pipelines other people’s decisions depend on. It covers five mechanisms: a contract at the boundary, writes that are safe to repeat, a freshness objective measured where data is consumed, schema changes absorbed rather than survived, and lineage used for triage. Each converts a silent failure into a loud one.

By the end you will be able to write assertions that catch the failures below, re-run any window without double counting, and answer the question a business owner actually asks, which is not “did the job run” but “is this number complete now”.

What you will be able to do

• Write a data contract stating grain, key, nullability, allowed lateness, and the action taken when a rule is violated.
• Make a batch or streaming write idempotent, so a replay of yesterday produces the same table as the original run.
• Define a freshness SLO a consumer understands, and alert on staleness rather than on job exit code.
• Distinguish late-arriving data from missing data, and restate a window instead of undercounting it.
• Name the six silent failure modes below by symptom, and place the assertion that catches each.

Why do broken data pipelines pass every health check?

Because most health checks measure the pipeline, not the data. Exit code zero, duration within range, no unhandled exception, cluster terminated cleanly: all four can be true while the target table gained nothing, gained the same rows twice, or lost a third of its rows to a join.

Orchestration monitoring answers “did the code run”. A scheduler cannot answer “is the result correct and complete” because it never looks inside the output, which is why teams find their worst incidents through a message from an analyst.

The fix is assertions that live inside the transaction and can fail the write. On Databricks, pipeline expectations do exactly that: a named boolean constraint on a dataset, with an action of warn, drop, or fail. The fail action rolls the update back atomically, so an invalid record prevents publication instead of reaching a consumer (Databricks expectations documentation).

CREATE OR REFRESH STREAMING TABLE shipments_silver(
  CONSTRAINT valid_key      EXPECT (shipment_id IS NOT NULL) ON VIOLATION FAIL UPDATE,
  CONSTRAINT valid_weight   EXPECT (gross_weight_kg BETWEEN 0 AND 200000) ON VIOLATION DROP ROW,
  CONSTRAINT plausible_date EXPECT (event_ts <= current_timestamp())
) AS SELECT * FROM STREAM(main.raw.shipments);

You know it is wired correctly when a malformed test record stops the update instead of appearing downstream.

What does a data contract have to contain to be worth writing?

A data contract is worth writing when it is executable, which means every clause maps to an assertion that can fail a run. A contract that lives only in a wiki is documentation, and documentation does not stop bad rows.

Five clauses carry almost all of the value. Grain: one row represents exactly what, stated as a sentence. Key: the columns unique at that grain, which is also the merge key. Nullability and domain per column, including enumerations, because a new status value is a schema-compatible change that silently breaks a CASE expression downstream. Allowed lateness: the delay after which an event is no longer accepted, which sizes your restatement window. Violation action per rule, decided in advance.

The Open Data Contract Standard gives you a portable YAML shape if you would rather not invent one (Open Data Contract Standard). The format matters less than the enforcement point. Put the contract at the boundary between raw landing and your first validated layer, and in the producer’s continuous integration if you can negotiate it, because a contract enforced only at the consumer end tells you the data is wrong after you stored it.

Decide early which violations quarantine and which fail. Failing on one malformed weight stops a hundred thousand good rows, and quarantining creates a queue somebody works. Quarantine anything recoverable, and fail on anything that would corrupt a key, a monetary amount, or a partition boundary.

How do you make a pipeline safe to re-run?

Make every write idempotent, meaning that running the same logical unit of work twice leaves the table in the same state as running it once. Without that property you cannot replay, and if you cannot replay you cannot fix anything without a repair script written under pressure.

Three mechanisms cover most cases. For a batch load, merge on the declared key rather than appending. For a full-window rebuild, use a dynamic partition overwrite scoped to the window. For streaming writes inside foreachBatch, Delta Lake accepts two writer options, txnAppId and txnVersion, and uses them to identify and ignore duplicate writes, so re-running an interrupted batch with the same pair is a no-op rather than a double insert (Databricks Structured Streaming with Delta Lake).

app_id = "shipments_silver_writer"

def write_idempotent(batch_df, batch_id):
    (batch_df.write.format("delta")
        .option("txnAppId", app_id)
        .option("txnVersion", batch_id)
        .mode("append")
        .saveAsTable("main.silver.shipments"))

stream.writeStream.foreachBatch(write_idempotent).start()

That mechanism has a sharp edge. Delta uses the batch identifier together with txnAppId as the unique key, and a new checkpoint restarts batch identifiers at zero, so if you delete a checkpoint and restart you must supply a different txnAppId. Otherwise the new run’s early batches are treated as already seen and skipped, producing a stream that runs cleanly and writes nothing.

Then make the replay unit, whether a day, an hour, or a source file, a parameter. Recovery should be a documented command, not an improvisation.

What does a freshness SLO look like in practice?

A freshness objective states the maximum acceptable age of the data where a consumer reads it, with an owner and an action when it is breached. “Refreshes hourly” is a schedule. “Ninety-five percent of the time, the newest row in gold.shipment_daily is no more than ninety minutes older than the event it describes” is an objective.

That requires three timestamps on every record, and most teams carry one. Event time is when the thing happened, arrival time is when your platform first saw it, publish time is when it became visible downstream. Only one of the lags below is yours to fix, and separating them ends the argument about whose fault the delay is.

Metric	Formula	Answers
Freshness	publish time minus event time	Can a consumer trust this number now
Pipeline lag	publish time minus arrival time	Is our processing keeping up
Source lag	arrival time minus event time	Is the upstream system late
Completeness	rows present versus rows expected for the window	Is the window whole or partial

Alert on freshness, not on job success. A job that succeeds hourly and produces nothing keeps freshness rising, so one alarm covers a failed run, a skipped run, an empty source, and a filter that excluded everything.

How do you stop late-arriving data being counted as missing?

Partition and aggregate by event time, keep a restatement window open, and never treat a count for a recent period as final. Late data is normal in any system with retries, offline devices, or manual entry, and a pipeline that closes the books at midnight undercounts every day.

In Structured Streaming the mechanism is a watermark. You declare an event-time column and a lateness threshold, and the engine keeps state long enough to fold in records arriving within it. The guarantee runs one way only: with a two-hour watermark the engine will not drop data less than two hours late, but data later than the threshold is not guaranteed to be dropped either, so it may or may not be aggregated (Spark Structured Streaming guide). A very late record can therefore change a number you already published.

(events
  .withWatermark("event_ts", "6 hours")
  .dropDuplicatesWithinWatermark(["event_id"])
  .groupBy(window("event_ts", "1 day"), "lane_id")
  .count())

For batch, the equivalent is a rolling restatement. Rebuild the last N days on every run, where N comes from the allowed lateness clause in your contract, and publish a completeness figure beside the number so a dashboard can mark yesterday as provisional. That prevents the daily chart with a permanent dip at its right edge, which analysts learn to ignore, and therefore also ignore on the day the dip is real.

How should schema changes be absorbed rather than survived?

Classify every change as additive, compatible, or breaking, and let only the first through automatically. Additive means a new nullable column every consumer can ignore. Compatible means a widening type change or a new enumeration value your logic already has a default branch for. Breaking means a removed column, a narrowed type, a renamed key, or a changed grain.

The trap is the middle class. A new value in a status column is schema-compatible and semantically breaking: the table absorbs it, the CASE WHEN that maps statuses to categories returns null, and a revenue category quietly loses rows. Enumerations therefore belong in the contract as an explicit allowed set, and an unexpected value should quarantine rather than become a null.

For breaking changes, version the contract and run both versions concurrently. Publish the new table beside the old, backfill it, move consumers one at a time, and retire the old one on a date. An in-place mutation converts one producer’s ten-minute decision into a dozen consumers’ unplanned afternoon.

What is lineage actually for?

Lineage is for two operations, impact analysis before a change and triage during an incident, and it earns its keep only if it is column-level and captured automatically. A hand-drawn dependency diagram is out of date the week after it is drawn.

Impact analysis asks “if I change this column, what breaks”. Triage asks the reverse: “this number is wrong, which upstream tables feed the column behind it, and which of them last changed”. Unity Catalog captures column-level lineage from query execution rather than from declaration, which is why it stays true as pipelines change. The test is behavioural: pick a Gold column, ask which raw tables feed it, and see whether you would act on the answer during an outage.

Where this goes wrong

The silently empty partition. A job finishes far faster than usual and writes zero rows, because a source path matched no files after an upstream export failed. Catch it with a row-count floor comparing the written count against a trailing median for the same weekday.

Late data counted as missing. The most recent bar dips and the monthly total grows after the month closes, because the pipeline aggregates by processing time and never restates. Fix it with event-time aggregation, a restatement window sized from the contract, and a completeness column.

A join that quietly drops rows. A fact table ends up smaller than its source with no error anywhere, because an inner join met a dimension that had not loaded. Use a left join plus an unmatched-key counter, with an expectation that output rows equal the driving side’s rows.

Deduplication on a key that is not unique. The row count drops a stable few percent after someone adds a dedup step, because dropDuplicates ran on a natural key that repeats once per status change. Assert uniqueness at the declared grain, so the pipeline fails rather than deleting the evidence.

Upstream soft deletes that never propagate. A cancelled record reports forever, because an incremental load filtered on a modified timestamp never sees a deletion recorded as a flag. Consume a change feed, or reconcile keys against a full snapshot on a schedule.

Local timestamps without an offset. One hour duplicates and one disappears, twice a year, because the source emits wall-clock local time. Store event time in UTC with the originating zone in its own column, and assert that the parsed timestamp is unambiguous.

Common questions

What is the difference between data quality and pipeline reliability?

Data quality describes properties of the data itself, such as validity, uniqueness, and completeness. Pipeline reliability describes whether the machinery delivers that data correctly and on time. They fail independently: a clean table can be eight hours stale, and a punctual pipeline can publish garbage. Assert on both in the same run, so one can block the other.

Should a data quality check fail the pipeline or just warn?

Fail when a violation would corrupt something expensive to unwind, such as a key, a monetary amount, or a partition boundary. Warn or quarantine when it affects individual records that can be reprocessed. That decision belongs in the contract, because during an incident the pressure is always to let the data through.

How do I detect an empty partition before a consumer does?

Compare the row count written against a trailing baseline for the same period and weekday, and fail the update when it falls below a stated fraction. Absolute thresholds age badly because volumes grow. A relative floor with a small absolute minimum catches both the zero-row case and the ninety-percent-loss case.

Does a data contract require a specific tool?

No. A contract needs a machine-readable definition of grain, key, nullability, domains, allowed lateness, and violation actions, plus a place where those clauses are enforced during a run. A checked-in schema file plus assertions in your first validated layer is a real contract.

How often should freshness be measured?

Continuously, from the consuming table rather than from the job. Compute the age of the newest record in the served table on a short interval and alert when it crosses the objective. That covers a failed run, a skipped run, an empty source, and a filter that excluded everything.

Next steps

Pick one table a business owner depends on and give it the full treatment this week: a written grain and key, three expectations with explicit violation actions, an idempotent write, and a freshness alarm on the served copy rather than on the job. One table done properly teaches more than a platform-wide policy nobody applies.

Then read data architecture patterns and how to choose by fit, because most reliability problems are a missing validated layer in a different costume, and where Databricks spend actually goes, which explains why re-reading raw data in five places is a correctness problem as well as a cost one. If documents rather than tables are your bottleneck, agentic document processing after OCR applies the same discipline to unstructured input.

Zorost builds these controls into client lakehouses as a trusted Databricks partner, where the parity test between the old report and the new one runs during the migration rather than after it. More engineering writing sits in the Signals index.