Databricks and PySpark | Full Course | - Vishal Mega Mart PySpark Interview Question

from pyspark.sql import SparkSession from pyspark.sql.functions import explode, split, col, trim,lower # Initialize Spark session spark = SparkSession.builder.getOrCreate() # Sample data data = [ (1, "Chair,Sofa,TV"), (2, "Chair,Chair,Sofa,tv,TV"), (3, "TV,TV,Sofa") ] # Create DataFrame df = spark.createDataFrame(data, ["ID", "Desc"]) df_1=df.withColumn("Desc_1",split(col("Desc"),",")) df_2=df_1.withColumn("Desc_2",(explode(split(col("Desc"),",")))) df_2=df_2.withColumn("Desc_2",lower(col("Desc_2"))) df_3=df_2.groupBy("Desc_2").count().withColumnRenamed("count", "count_desc").withColumnRenamed("Desc_2", "desc") display(df_3) 1. Azure End to End Projects -https://www.geekcoders.co.in/courses/Build-End-to-End-Azure-Data-Engineering-Project-644fba73e4b0a1072ff43a72-644fba73e4b0a1072ff43a72 2. Databricks Certified Course -https://www.geekcoders.co.in/courses/Databricks-Certified-Data-Engineer-Zero-to-Hero-65eea8c0940e6c57bba246f7 3. PySpark Interview Questions -https://www.geekcoders.co.in/courses/Practice-50-PySpark-Interview-Questions-65f317e33210a77d47c6afaf 4. Full Course - https://www.geekcoders.co.in/courses/Learn-Python-Build-Azure-End-End-Project-Deep-Dive-in-Delta-with-project-RealTime-Scenarios-of-AzureDataFactory-Databricks-and-PySpark-64a17173e4b0b85ec6ffe0eb-64a17173e4b0b85ec6ffe0eb 5. Microsoft Fabric: https://www.geekcoders.co.in/courses/Learn-Microsoft-Fabric-Basic-to-Advance-with-Project-659676e4e4b03dbe2303663b Explore other courses too - https://www.geekcoders.co.in/s/store 🔹 **What You’ll Learn:** - Setting up and managing **Databricks** for big data analytics and machine learning. - Creating and orchestrating ETL pipelines with **Azure Data Factory**. - Writing efficient queries and managing databases with **SQL**. - Leveraging **Python** for data transformation and automation. - Building interactive dashboards and visualizations using **Power BI**. 🔹 **Why Watch?** - Gain hands-on experience with real-world data engineering scenarios. - Learn how to integrate Azure tools for seamless workflows. - Prepare for **Azure Data Engineer certifications** and career advancement. Whether you're a beginner or looking to enhance your skills, this course provides step-by-step tutorials, practical examples, and pro tips to help you succeed. Don’t forget to **like, share, and subscribe** for more tech content! 👍 #Azure #DataEngineering #Databricks #AzureDataFactory #PowerBI #Python #SQL #genai #azuredatafactory