Data Engineering on Microsoft Azure - Azure for DE: 14 - Error handling in Azure Data Factory

Hey data engineers! Let's be realistic - everyone makes bugs (including you and me) so your ADF pipelines should be able to handle them. Also, when developing solutions in a cloud, we should be prepared to deal with transient issues. In this video I'm talking about various stuff related to ADF error handling and monitoring: - How to handle transient errors - What are the common error handling patterns and how to implement them - How to turn on alerting in just 30 minutes - How to create a notification part that will send you an e-mail if something bad happens (including customized e-mail body) - How to make sure that pipeline logs can be stored and queries for a longer period than just 45 days Enjoy! ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 ADF Errors and Conditional execution: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling Send an email with an Azure Data Factory: https://learn.microsoft.com/en-us/azure/data-factory/how-to-send-email ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:57 Retry 04:10 Conditional paths 10:07 Determining pipeline status 12:46 Common error handling patterns 29:14 Parent - child scenarios 38:33 Alerts 47:35 Logic app approach 1:01:42 Log analytics workspace approach 1:16:39 Summary