Data Engineering on Microsoft Azure - Azure for DE: 37 - Orchestrating Databricks notebooks

Hey, data engineers! We have successfully developed Databricks notebooks for data transformation, and they are functioning smoothly. However, up to this point, we have been executing them manually, cell by cell. How can we automate this process and integrate it into our orchestration workflow? Discover the solution by tuning in to the 37th episode of my free DP-203 series. Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Databricks pricing: https://azure.microsoft.com/en-us/pricing/details/databricks/ MS tutorial to run notebook from ADF: https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook Compute permissions in Databricks: https://learn.microsoft.com/en-us/azure/databricks/compute/clusters-manage#--compute-permissions ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:28 Scenario 04:06 Sample notebook 05:59 Sample ADF pipeline 09:10 Cluster types 13:45 Authentication types 21:50 Using ADF managed identity 40:27 Using a job cluster 51:44 Parametrizing the notebook 56:53 Using Databricks jobs 1:01:18 Summary