Google Cloud End to End Data Engineering Projects - Google Cloud Realtime Streaming | End to End Data Engineering Project

Master End-to-End Data Engineering: Real-Time Streaming, AI Integrations, and High-Performance Systems! Dive into hands-on projects, expert-guided tutorials, and cutting-edge technologies for a standout career in data engineering.

4.0 (2)

25 learners

What you'll learn

Understand and implement real-time streaming with Google Cloud for data engineering projects
Learn how to perform real-time socket streaming using Apache Spark
Master the use of Apache Airflow alongside Spark, Pyspark, Java, and Scala for data engineering
Develop skills to build and optimize high-performance, real-time analytics databases

This course includes

47.5 hours of video
Certificate of completion
Access on mobile and TV

Summary

Keywords

Python Docker Kafka Zookeeper Schema Registry Connect ksqlDB Telegram Data Engineering Real-Time Analytics YouTube API YouTube Analytics

Full Transcript

In this comprehensive tutorial, you will build an end-to-end data engineering pipeline for real-time YouTube Analytics. Each time there's an activity on any video you or playlist of your choice, you get instant notification on Telegram. 📋 What You Will Learn: ✅ How to fetch data from YouTube API using Python ✅ Setting up a Kafka ecosystem using Docker and Confluent containers ✅ Processing and streaming data using ksqlDB ✅ Sending data to external systems with connectors ✅ Real-time notifications on Telegram ✅ Python Advanced Concepts ✅ Google cloud configuration for Youtube 🛠 Technologies Used: ✅ Python ✅ Google Cloud ✅ Docker ✅ Telegram 🔗 Useful Links: ✅ GitHub Repo for this Project: https://github.com/airscholar/YoutubeAnalytics.git ✅ Confluent Official Documentation: https://docs.confluent.io/home/overview.html ✅ YouTube API Documentation: https://developers.google.com/youtube/v3/docs ✅ Kafka-Python Documentation: https://kafka-python.readthedocs.io/en/master/ 🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟 🗨 Comments & Questions Have questions or ran into issues? Drop a comment below and I'll do my best to help you out! TIMESTAMPS: 0:00 Introduction 2:21 Setting up the system architecture on Docker 17:46 Control Center Demo 23:20 Getting Youtube API Key from Google Cloud 27:21 Fetching Data From Youtube with Python 37:53 Streaming Data to Kafka 45:13 Advanced Python Concept 59:31 Stream Processing with KSQLDB 1:09:50 Setting up Telegram Bot 1:13:08 Connecting to external systems from Kafka (Telegram) 1:27:18 Outro ✨ Tags ✨ Data Engineering, Kafka, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Schema Registry, Control Center, Data Streaming, Google Cloud, Youtube API, Telegram, KSqlDb, Confluent Connect ✨ Hashtags ✨ #confluent #DataEngineering #ApacheAirflow #Kafka #telegram #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics #docker #python #java