Data Engineering on Microsoft Azure - Azure for DE: 44 - Loading data to Dedicated SQL Pool

Hey data engineers! If you’ve chosen to use Dedicated SQL Pool for your data storage and serving needs, you might be wondering how to load data into it. What options are available, and what are their pros and cons? Join me for the 44th episode of my free DP-203 course, where I explore various methods for inserting data into Dedicated SQL Pool: • PolyBase • COPY INTO • ADF/Synapse data flows • ADF/Synapse copy activity • Synapse Spark Pools • Databricks notebooks Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Data ingestion overview: https://learn.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/design-elt-data-loading?context=%2Fazure%2Fsynapse-analytics%2Fcontext%2Fcontext Design a PolyBase data loading strategy: https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/load-data-overview COPY INTO: https://learn.microsoft.com/en-us/sql/t-sql/statements/copy-into-transact-sql?view=azure-sqldw-latest Dedicated SQL Pool connector for Apache Spark: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-spark-sql-pool-import-export?tabs=scala%2Cscala1%2Cscala2%2Cscala3%2Cscala4%2Cscala5 Query Dedicated SQL Pool from Databricks: https://learn.microsoft.com/en-us/azure/databricks/connect/external-systems/synapse-analytics ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:25 Using Polybase 33:09 Using COPY 38:11 Using data flow 55:46 Using Copy Activity in ADF 1:04:55 Using Spark Pools 1:12:34 Using Databricks 1:18:08 Summary