26 Spark SQL, Hints, Spark Catalog and Metastore | Hints in Spark SQL Query | SQL functions & Joins

PySpark - Zero to Hero | PySpark Tutorial 2025 | Spark Tutorial 2025 | Learn from Basics to Advanced Performance Optimization - 26 Spark SQL, Hints, Spark Catalog and Metastore | Hints in Spark SQL Query | SQL functions & Joins

4.0 (1)

18 learners

What you'll learn

This course includes

9 hours of video
Certificate of completion
Access on mobile and TV

Summary

Keywords

spark interview questions apache spark tutorial beginners apache spark python apache spark full course how does apache spark work apache spark performance optimization techniques optimize pyspark Spark catalyst optimizer tungsten Catalyst optimizer interview questions Wide transformations spark examples Repartition coalesce pyspark Predicate pushdown spark example

Full Transcript

Video explains - How to use SQL with Spark? How to use Persisting Metastore ? What is Hive Catalog in Spark? Chapters 00:00 - Introduction 00:47 - Spark SQL Demonstration 01:42 - What is Spark SQL Catalog? 03:56 - Temp View in Spark SQL 06:25 - Using SQL function in queries 08:59 - Joins in Spark SQL 10:41 - Hints in Spark SQL Queries 12:19 - Save as Table in Spark 13:02 - Read data in Spark SQL 15:05 - Persisting Metadata with Hive Metastore 18:27 - Show metadata of tables Local PySpark Jupyter Lab setup - https://youtu.be/WhxljT3IfdM Python Basics - https://www.learnpython.org/ GitHub URL for code - https://github.com/subhamkharwal/pyspark-zero-to-hero/blob/master/21_aqe_spark.ipynb The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing. New video in every 3 days ❤️ #spark #pyspark #python #dataengineering