Data Engineering on Microsoft Azure - Azure for DE: 36 - Automating the process with Azure Databricks Autoloader

Hey data engineers! Ever wished for automated processing of new data files as they're ingested into ADLSg2? Well, it's not just wishful thinking – it's entirely achievable! Simply configure Azure Databricks Autoloader. Come aboard for the 36th episode of my free DP-203 course, where I'll guide you through the setup and functionality of Autoloader. Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Documentation about Autoloader: https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/ ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:29 Our scenario 05:46 Environment setup 13:09 Autoloader in action 18:04 Schema inference 23:14 Schema evolution 34:26 Error handling 36:25 Batch processing 38:54 File detection modes 44:06 Summary