Google Cloud End to End Data Engineering Projects - Realtime Voting System | End-to-End Data Engineering Project

Master End-to-End Data Engineering: Real-Time Streaming, AI Integrations, and High-Performance Systems! Dive into hands-on projects, expert-guided tutorials, and cutting-edge technologies for a standout career in data engineering.

4.0 (2)

25 learners

What you'll learn

Understand and implement real-time streaming with Google Cloud for data engineering projects
Learn how to perform real-time socket streaming using Apache Spark
Master the use of Apache Airflow alongside Spark, Pyspark, Java, and Scala for data engineering
Develop skills to build and optimize high-performance, real-time analytics databases

This course includes

47.5 hours of video
Certificate of completion
Access on mobile and TV

Summary

Keywords

Data Engineering Apache Airflow Kafka Apache Spark PostgreSQL Zookeeper Docker Docker Compose ETL Pipeline Data Pipeline Big Data Streaming Data

Full Transcript

In this video, we'll build an end to end realtime voting system with the use of modern bidata technologies like Apache Kafka, Apache Spark and Streamlit. Timestamps: 0:00 Introduction 1:06 The Election Flow 2:00 The System Architecture 3:02 Setting up the Project 5:16 The docker compose for the system architecture 10:10 Candidate, Party and Voters Accredditation 45:05 Voting System 1:02:03 Streaming votes from/to Kafka 1:10:43 Reading Voting Data Streams from Kafka with Pyspark 1:13:39 Spark Data preprocessing 1:15:54 Spark Aggregations 1:18:58 Writing aggregated data back to kafka 1:28:43 Realtime Votes Dashboard with Streamlit 2:14:19 Results and Live commentaries 2:16:16 Winner of the Election 2:17 31 Outro 👍 Support Our Channel: If you find our content valuable, consider supporting us by joining our channel membership, where you'll get exclusive access to behind-the-scenes content, Q&A sessions, and much more! https://www.youtube.com/channel/UCAEOtPgh29aXEt31O17Wfjg/join Resources and Links Full code: https://github.com/airscholar/realtime-voting-data-engineering Medium Article: https://medium.com/@yusuf.ganiyu/building-a-real-time-election-voting-system-an-end-to-end-data-engineering-project-db372d0f5a39 Streamlit Documentation: https://docs.streamlit.io/ More Videos Like this: datamasterylab.com 💬 Join the Conversation: We love hearing from you! Share your thoughts, questions, or experiences related to data engineering or this project in the comments below. Don't forget to like, subscribe, and hit the bell icon to stay updated with our latest content. 🙏 Thank You for Watching! Remember to subscribe and hit the bell icon for notifications. Stay curious and keep exploring the fascinating world of data engineering! #RealTimeVotingSystem #DataEngineering #ElectionTechnology #BigData #DataScience #SoftwareDevelopment #TechTutorial #Educational #ProjectShowcase #DataEngineeringProject #Pyspark #Streamlit #Kafka

Continue this lesson in the app

Install CourseHive on Android or iOS to keep learning while you move.

Related Courses

30-Day Beginner Guitar Challenge for New Players

Master the Guitar in 30 Days: Your Ultimate Beginner Challenge! Unleash your inner guitarist with step-by-step lessons designed to transform you from novice to confident player. Join Your Guitar Academy and kickstart your musical journey today!

⭐ 4.3

36 ratings

7 hours