Data Engineering on Microsoft Azure - Azure for DE: 32 - A closer look at Databricks notebooks

Hey data engineers! Are you interested in exploring Databricks notebooks more closely with me? For instance, we can examine how simple it is to switch between languages or exchange data seamlessly. Additionally, I'll share a real-life story about a surprise I encountered due to overlooking the significance of pinning a cluster. We'll also discuss methods for practicing data transformations, even in the absence of any data. If you're intrigued by these topics, feel free to join me in the 32nd episode of my free DP-203 course. ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Azure Databricks princing: https://azure.microsoft.com/en-us/pricing/details/databricks/ Pin compute: https://learn.microsoft.com/en-us/azure/databricks/compute/clusters-manage#--pin-a-compute ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:15 Pricing tiers 04:04 Pin cluster 07:07 Multiple clusters 09:05 Workspace menu 15:01 Hello world notebook 19:03 Dbutils 22:21 Sample datasets 32:42 Databricks managed RG 39:21 Loading our CSV file 57:50 Loading CSV file using SQL 59:31 Exchanging data between languages 1:06:35 Doing some transformations 1:11:55 Visualisations 1:14:00 Summary