Data Engineering on Microsoft Azure - Azure for DE: Processing new and changed data with MERGE

Master Azure Data Engineering: From Ingestion to Analytics with Real-World Projects

4.0 (5)

45 learners

What you'll learn

Design and implement a data ingestion pipeline using Azure Data Factory
Apply security and access controls to an Azure Data Lake Storage account
Transform raw data into a modeled structure using Azure Databricks or Synapse Spark
Deploy and manage Azure Data Factory pipelines using CI/CD practices

This course includes

49.5 hours of video
Certificate of completion
Access on mobile and TV

Summary

Keywords

azure data engineering incremental processing merge statement incremental load data engineering patterns upsert pattern etl design azure databricks delta lake merge enterprise data engineering MERGE

Full Transcript

Hey data engineers! You've identified new and changed data. Now what? How do you insert and update it in your target table in a clean, scalable way? MERGE. This episode focuses on the processing phase of the universal incremental pattern and shows how to use the MERGE statement. Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ My GitHub with drawings and source code (available only to Senior Data Engineer members): https://github.com/TybulOnAzure/Senior-Data-Engineer ▬▬▬▬▬▬ SUPPORTING MY CHANNEL ▬▬▬▬▬▬ Buy me a coffee (or beer): https://buymeacoffee.com/piotrv Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:28 MERGE explained 05:44 MERGE in action 20:53 Performance optimization for MERGE 37:24 Summary

Continue this lesson in the app

Install CourseHive on Android or iOS to keep learning while you move.

Related Courses

30-Day Beginner Guitar Challenge for New Players

Master the Guitar in 30 Days: Your Ultimate Beginner Challenge! Unleash your inner guitarist with step-by-step lessons designed to transform you from novice to confident player. Join Your Guitar Academy and kickstart your musical journey today!

⭐ 4.3

36 ratings

7 hours