Data Engineering on Microsoft Azure - Azure for DE: 41 - Transforming data with Data Flows

Hey data engineers! In our work, we often need to transform ingested data. So far, I've introduced you to three tools for this purpose: Azure Databricks notebooks, Synapse Analytics notebooks and dbt. However, all of them require you to write code using SQL, Python, Scala, or R. But what if you don't know any of these languages and still need to transform data? Is it possible? Yes, it is! You can use Data Flows to visually transform your data without writing any code. Join me in the 41st episode of my free DP-203 course, where I discuss Data Flows and demonstrate how to use them to transform our Rebrickable data. ▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬ My LinkedIn profile: https://www.linkedin.com/in/piotr-tybulewicz-81a8793/ GitHub with my drawings: https://github.com/TybulOnAzure/DP-203 Comparison of IRs: https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration All transormations: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-transformation-overview ▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬ Join this channel to get access to perks: https://www.youtube.com/channel/UCLnXq-Fr-6rAsCitq9nYiGg/join ▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬ 00:00 Introduction 00:35 What are data flows? 03:23 Creating a data flow 05:39 Adding a Source 17:54 Flattening the array 24:07 Handling NULLs 33:12 Removing and renaming columns 35:48 Converting data types 37:43 Adding conditional logic 41:19 Saving transformed data to a data lake 45:12 Executing dataflows 49:39 Quick overview of all transformations 57:33 Summary