Data Engineering on Microsoft Azure - Azure for DE: 35 - Writing data to ADLSg2 from Azure Databricks

Greetings, data engineers! You've successfully transformed your data with Azure Databricks, but what's next? How do you ensure its persistence? How can you prevent your results from disappearing once you've terminated the Databricks cluster? Join me in the 35th episode of my free DP-203 course, where I discuss different methods for saving data from Azure Databricks to ADLSg2, including: • Managed tables • External tables • Saving data without registering any table Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Recommendations for working with DBFS root: https://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 01:03 Cluster configuration 04:02 Preparing the data 05:24 Saving the data to ADLSg2 10:19 Saving the data as managed tables 25:05 Saving the data as external tables 30:18 Uploading new data 32:13 Database vs schema 35:15 Querying our data 41:48 Identity column 45:46 Summary