Google Cloud End to End Data Engineering Projects - Realtime Stock Market Anomaly Detection using ML Models | An End to End Data Engineering Project

In this video, you will be developing an end-to-end data engineering project focused on Anomaly Detection using state-of-the-art tools like Quix Streams, Redpanda, and Docker. We'll guide you step by step through building a complete data pipeline, from collecting stock market data to deploying an advanced anomaly detection model. You'll gain hands-on experience with real-time data streaming, model building with Isolation Forests, and containerizing your project for seamless integration. This project is perfect for anyone looking to level up their skills in modern data engineering and machine learning! What You Will Learn: ✅ How to architect a complete anomaly detection pipeline ✅ Methods for fetching stock market data via FTP ✅ Setting up a stock data producer with Quix Streams ✅ Creating and deploying an anomaly detection system using Isolation Forests ✅ Real-time data transformation and processing with Quix Streams ✅ How to integrate Docker for streamlined project deployment ✅ Tips on troubleshooting and resources from Quix documentation Timestamps: 0:00 Introduction 1:07 System Architecture 2:44 Getting Stock Market Data via FTP 6:19 Setting Up a Fresh Project 11:09 Creating a Stock Market Data Producer with Quix Starter Source 26:40 Creating an Anomaly Detector using Quix Transformation Source 40:43 Building an Isolation Forest Model for Anomaly Detection 56:34 Testing and Review of Results 58:50 Quix Documentation and Help Resources 1:01:26 Outro 🔗 Useful Links and Resources: ✅ Source Code: https://github.com/airscholar/RealtimeAnomalyDetection ✅ Article: https://link.medium.com/gtgR3WMjnNb ✅ QuixStreams official documentation: https://quix.io/docs/ ✅ Join Quix Community and get swift help: https://quix.io/slack-invite ✅ Docker Compose Documentation: https://docs.docker.com/compose/ ✨ Hashtags ✨: #DataEngineering #QuixStreams, #AnomalyDetection #BigData #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics #EndtoEndDataEngineeringProject