Google Cloud End to End Data Engineering Projects - Decodable vs AWS Managed Service for Apache Flink (MSF) - End to End Data Engineering Project

In this video, the ease of use between Decodable and Amazon Managed Service for Apache Flink (MSF) is compared with an end-to-end data engineering project. You will build the projects and experience how both platforms handle real-time streamind data from Kinesis Streams, performing transformations, aggregations, and sinking results into PostgreSQL, S3, Apache Druid, Delta Lake, and Elasticsearch. You will experience how Decodable simplifies pipeline creation with its intuitive interface, versus the more complex setup required for AWS MSF. This comparison will help you decide which platform suits your streaming data needs best! Timestamps: 00:00 Introduction 1:45 Systems architecture 4:15 Setting up a new Project and Installing Dependencies 11:06 Coding the Sensor Data Producer 25:21 Setting up Kinesis Streams 27:00 Setting up your Decodable Account 29:00 Setting up Connection to AWS Kinesis from Decodable 35:40 Connecting to Kinesis Streaming Sources from Decodable 40:45 Creating Pipelines on Decodable 44:48 Creating Transformation and Aggregation Pipelines on Decodable 59:00 AWS Managed Service for Apache Flink (MSF) 1:00:30 Setting up a new Apache Flink MSF Project 1:19:31 Connecting to Kinesis Streams from Apache Flink 1:36:28 Transforming Kinesis Stream Data with Apache Flink 1:37:35 Writing Data to Postgres Sink with Apache Flink 2:06:15 Packaging Flink Code for AWS MSF 2:15:00 Creating AWS Managed Service for Apache Flink Cluster 2:17:00 Submitting Flink Job to MSF Cluster 2:20:00 Fixing Bugs, Updating Code Snapshots, Repackaging and Resubmission 2:30:10 Wrapping up 2:31:21 Documentation, Resources and Getting help 2:33:00 Outro Resources and Links: ✅ Free tier sign-up: https://dcdbl.co/3MqqDf7 ✅ Quickstart guide to accelerate your real-time ETL, ELT, and streaming initiatives: https://docs.decodable.co/get-started/quickstart ✅ Learn more about Decodable in our Architecture Guide: https://dcdbl.co/3Xpfhyk Full Flink Source Code: buymeacoffee.com/yusuf.ganiyu/decodable-vs-amazon-managed-service-apache-flink-msf-end-end-de-project ✅ Prefer to read: https://medium.com/towards-data-engineering/decodable-vs-aws-managed-service-for-apache-flink-msf-an-end-to-end-data-engineering-showdown-628ecf0428a8 ✨Tags: Data Engineering Project, Decodable, Amazon Managed Service for Apache Flink, Big Data, Apache Flink, AWS MSF, Data Engineering, Real-time Data Streaming, Kinesis Streams, PostgreSQL, S3, Apache Druid, Delta Lake, Elasticsearch, Data Transformation, Data Aggregation, End-to-End Data Engineering Project, Streaming Data Pipelines, ETL, ELT, Data Integration, AWS Kinesis, Cloud Data Engineering, Data Analytics, Data Processing ✨ Hashtags: #DataEngineering, #Decodable, #ApacheFlink, #AWSMSF, #StreamingData, #RealTimeData, #KinesisStreams, #ETL, #ELT, #DataPipelines, #BigData, #CloudComputing, #AWS, #DataTransformation, #DataIntegration, #TechTutorial, #CloudServices, #DataAnalytics