Summary
Keywords
Full Transcript
Video explains - How Spark distributes memory? What is Spark Memory Management? If Spark can Splill data then why OOM Error? What is JVM On-Heap and Off-Heap Memory? What is the minimum amount of Memory needed for Executors? What is Spark Unified Memory? What are Storage and Execution Memory? What is OOM Error? WHat are the reasons for OOM Errors? What is Garbage Collection? What is Serialization and Deserialization? Chapters 00:00 - Introduction 00:39 - Spark Memory Management in Theory 01:39 - JVM On-Heap vs Off-Heap Memory 03:51 - Reserved Memory 04:24 - Minimum amount of Memory for Executors in Spark 05:26 - User Memory 07:05 - Spark Unified Memory - Storage and Execution Memory 12:53 - Storage Memory Spill 15:40 - Spillage during Execution 19:25 - Understand OOM Errors 23:00 - Serialization and Deserialization 23:59 - Off Heap Memory 25:59 - Garbage Collection(GC) 27:35 - Out Of Memory Errors in Practical 28:09 - Spark Cluster Setup in Local Code at Github - https://github.com/subhamkharwal/pyspark-zero-to-hero/blob/master/27_spark_memory_and_oom.ipynb Files for OOM errors, zipped in single zip file - https://github.com/subhamkharwal/pyspark-zero-to-hero/blob/master/spark_oom_files.7z GitHub URL for docker images for cluster setup - https://github.com/subhamkharwal/docker-images The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.
Continue this lesson in the app
Install CourseHive on Android or iOS to keep learning while you move.
