+91 98726 60544 hello@mitstech.co Mon–Sat · 09:00–18:30 IST

Data catalogues and lineage: knowing where a number came from

Data By Mits Data Practice 1 min read
Data catalogues and lineage: knowing where a number came from

Every organisation with more than a few data pipelines reaches the same moment: two reports show different revenue figures, and resolving it takes days of tracing transformations backwards through jobs nobody has looked at in a year.

Lineage is the record of how data moved and what was done to it. Catalogue is the record of what exists and what it means. They solve different halves of the same problem, and either alone leaves you partly blind.

The cheapest version of lineage is not a tool. It is discipline: every transformation defined in version-controlled code rather than in a scheduler's web interface, and every table documented with the query that produces it. Frameworks such as dbt give this almost free, because the dependency graph is derived from the code itself.

A catalogue earns its place when the number of tables exceeds what a person can hold in their head, which happens sooner than expected. What matters is not the tool's feature list but whether anyone maintains it. A catalogue with stale descriptions is worse than none, because it is confidently wrong.

Ownership is the field that makes a catalogue useful. Not a team name — a person, who can answer a question about that table. Most catalogue projects fail because they document structure and skip ownership, and structure alone does not settle an argument about a number.

There is a compliance dividend. Lineage tells you which downstream systems hold personal data derived from a source table, which is exactly what you need when someone exercises a deletion right under the DPDP framework. Teams that built lineage for analytics reasons find they have solved a privacy problem too.

Back to all news

Keep reading

More on Data