Summary
Keywords
Full Transcript
In this hands-on project, learn how to set up real-time monitoring for a high-performance data pipeline processing 1.2 billion records per hour! We cover everything from configuring Grafana and Prometheus to creating a comprehensive Kafka and Spark monitoring dashboard. Watch to see how modern data engineering tools come together for effective system monitoring. PART 2: https://youtu.be/PL6Tl2sqh8k Timestamps: 0:00 Introduction 2:25 Reading and Parsing Logs from Different Source 4:17 Setting up Elasticsearch for High Throughput 7:45 Elasticsearch UI Navigation 9:02 Setting up Filebeat 20:46 Setting up Logstash 28:00 Querying Indexed Records on Elasticsearch 30:00 Outro Like this video? Support us: https://www.youtube.com/@CodeWithYu/join 👀 Don't just watch it, build it! 🚧 👍 Like, Comment, & Subscribe for more cutting-edge data engineering content! Resources: Full Source Code: https://buymeacoffee.com/yusuf.ganiyu/full-source-code-monitoring-high-performance-architecture-systems Kafka Documentation: https://kafka.apache.org/documentation/ Apache Spark Documentation: https://spark.apache.org/documentation.html JMX Exporter Agent - https://github.com/prometheus/jmx_exporter/releases #Elasticsearch, #Logstash, #Filebeat, #Kibana, #DataEngineering, #BigData, #RealTimeProcessing, #BigDataAnalytics, #DataPipeline, #StreamingData, #KafkaMonitoring, #SparkStreaming, #DataArchitecture, #HighPerformanceComputing
Continue this lesson in the app
Install CourseHive on Android or iOS to keep learning while you move.
