Data Engineering on Microsoft Azure - Azure for DE: 08 - Common file types (Delta lake)

Hey data engineers! Join me in exploring the game-changer in the world of big data analytics - Delta Lake. Ever wondered how to ensure data reliability, ACID transactions, and time travel, all in one package? Find out why Delta Lake is the ultimate storage format that's revolutionizing the industry. ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Blog with samples used: https://delta.io/blog/2022-11-16-delta-lake-schema-enforcement/ ▬▬▬▬▬▬ SUPPORTING MY CHANNEL ▬▬▬▬▬▬ Buy me a coffee (or beer): https://buymeacoffee.com/piotrv Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:33 Delta lake overview 03:10 Delta lake structure 16:33 ACID guarantees 31:29 Audit history 33:40 Time travel 38:24 Vacuum command 42:00 Schema enforcement 47:20 Checks and constraints 49:33 Schema evolution 52:02 MERGE operation 54:49 Optimize and z-order 57:34 Unified batch and streaming 58:41 Summary