Data Engineering on Microsoft Azure - Azure for DE: 30 - Slowly changing dimensions, data lake structure

Hey data engineers! In an ideal scenario, our dimensions would remain static. However, in reality, they undergo changes which we must address. Thankfully, there are strategies for managing this, known as slowly changing dimensions. Additionally, it's crucial to acknowledge that our data must undergo transformation from its raw and messy state into structured facts and dimensions. Yet, the question arises: how many layers should comprise our data lake? Is adherence to the medallion architecture always advisable? Join me in the 30th episode of my free DP-203 course, where I cover these topics. ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Medallion architecture: https://www.databricks.com/glossary/medallion-architecture Cloud-scale analytics: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/best-practices/data-lake-zones Discussion on LinkedIn: https://www.linkedin.com/posts/sqlbi_medallion-databricks-fabric-activity-7140269823653044224-Tg4c?utm_source=share&utm_medium=member_desktop Behind the Hype - The Medallion Architecture Doesn't Work: https://www.youtube.com/watch?v=fz4tax6nKZM ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:48 Slowly changing dimensions 04:38 SCD Type 1 08:30 SCD Type 2 22:45 SCD Type 3 28:40 Data lake design 33:21 Medallion architecture 39:30 Microsoft approach 47:26 Summary