Summary
Keywords
Full Transcript
Video explains - How to read CSV Files? How Spark works in background while reading files? Understand Spark UI. What is InferSchema? What are the read modes available in Spark? Chapters 00:00 - Introduction 01:29 - How to read CSV file in Spark 02:29 - What is happening in Spark UI ? 03:15 - Read header from file 04:00 - Spark InferSchema Option 06:02 - Read file with Schema 08:27 - CSV File read options 08:54 - Read Modes 09:25 - Permissive Mode 13:14 - Drop Malformed Mode 14:23 - Fail Fast Mode 15:35 - Use case for Read Modes 16:05 - Bonus Tip Spark CSV Documentation - https://spark.apache.org/docs/latest/sql-data-sources-csv.html Local PySpark Jupyter Lab setup - https://youtu.be/WhxljT3IfdM Python Basics - https://www.learnpython.org/ GitHub URL for code - https://github.com/subhamkharwal/pyspark-zero-to-hero/blob/master/08_reading_from_csv_files.ipynb The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing. New video in every 3 days ❤️ #spark #pyspark #python #dataengineering
Continue this lesson in the app
Install CourseHive on Android or iOS to keep learning while you move.
