Data pipelines have a failure mode software does not: succeeding while being wrong. The job runs, the rows load, the dashboard refreshes, and the numbers are incorrect because an upstream system changed a field or a join silently duplicated rows. Nothing errors. The discovery happens days later, in a meeting, when someone notices a figure that cannot be right.
The defence is to assert expectations about the data itself, at each stage, and fail the pipeline when they are violated. Row counts within an expected range - a source that usually delivers a hundred thousand rows and today delivered four hundred has a problem, whether or not it errored. Nulls in columns that must not be null. Uniqueness of primary keys. Referential integrity across the join you just performed. Values within plausible bounds. These are the data equivalent of unit tests and take minutes to write.
The single highest-value assertion is a uniqueness check after every join, because the most common silent corruption in analytics is a join that fans out. A one-to-one join against a table that unexpectedly contains duplicates quietly doubles your revenue figures, and every downstream aggregate inherits it. Asserting that the row count did not change when you expected it not to catches this immediately rather than at quarter-end.
Freshness deserves its own monitor, separate from correctness. A pipeline that failed to run at all produces no error - it produces silence, and the dashboard keeps showing yesterday's numbers looking perfectly healthy. Alert on the age of the most recent record, per source, rather than only on job failures, because absence is the failure you cannot see.
Where you place the checks matters. Validate on ingestion, so bad data is caught at the boundary and can be attributed to a source system rather than discovered three transformations later. Validate again after transformation, where your own logic is the suspect. And decide per check whether a violation should halt the pipeline or warn - halting on a critical financial table is right; halting the entire warehouse because one optional column has more nulls than usual is how teams end up disabling the checks.
Finally, treat data incidents like software incidents. Who is on the hook when the numbers are wrong, how are consumers told, and what stops it recurring. The dashboard that quietly showed incorrect figures for a week does more damage to trust than the pipeline that failed loudly on Monday, because after the first one nobody believes any of the numbers - and rebuilding that belief takes considerably longer than fixing the pipeline.
