Data Engineering on Microsoft Azure - Data lineage in practice: Unity Catalog and Microsoft Purview

Hey data engineers! Ever wondered which tables feed a report when the numbers suddenly look wrong? Or who is actually using your data right before you are about to introduce a breaking change? That's exactly where data lineage becomes a lifesaver. In this episode, I explain what data lineage really is, why it matters in real projects, and when it can save you from serious trouble. I also walk through a complex, real-life architecture and show how lineage is captured using Unity Catalog and Microsoft Purview. Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ My GitHub with drawings and source code (available only to Senior Data Engineer members): https://github.com/TybulOnAzure/Senior-Data-Engineer ▬▬▬▬▬▬ SUPPORTING MY CHANNEL ▬▬▬▬▬▬ Buy me a coffee (or beer): https://buymeacoffee.com/piotrv Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:32 What is data lineage? 04:28 Why do we need data lineage? 06:35 Real-life example 14:52 Lineage in Unity Catalog 29:19 Lineage in Microsoft Purview 38:15 Summary