Data Engineering on Microsoft Azure - Azure for DE: 40 - Extra features in Azure Synapse Analytics Spark Pools

Hey, data engineers! You're already aware that Spark Pools in Azure Synapse Analytics can seamlessly integrate with various Azure services, such as ADLSg2. But what about other services? Join me in the 40th episode of my free DP-203 course, where I answer the following questions: • How is it possible for a Spark Pool to retrieve data from ADLSg2? • Will it work out of the box with any other storage account? • Will it behave differently when deployed in production? • How can you execute a notebook from a Synapse pipeline? • How can you parameterize a Synapse notebook? • How can you retrieve a secret from Azure Key Vault using a Synapse notebook? Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Docs: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:29 Connecting to ADLSg2 12:32 Using managed identity 15:53 Using linked service 20:57 Calling notebook from pipeline 28:31 Parametrizing notebooks 35:48 Synapse vs ADF vs Databricks 40:16 Integration with Key Vault 48:49 Data partitioning 57:33 Summary