Imported from Galius5136/databricks-spark-3.5-cert-prep (skills/apache-spark/SKILL.md). Install upstream with npx skills add Galius5136/databricks-spark-3.5-cert-prep --skill apache-spark. Copyright stays with the author.

Learning Spark, 2nd Edition — Knowledge Base

Authors: Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee | Chapters: 12 | Generated: 2026-05-24

This skill is tuned for Databricks Certified Associate Developer for Apache Spark exam prep — PySpark first, with emphasis on architecture, DataFrame API, Spark SQL, tuning, and Structured Streaming.

How to Use This Skill

• Without arguments — load the Core Frameworks below as a Spark mental model.
• By topic — ask about shuffle partitions, broadcast join, watermark, output modes, etc. → I find and read the relevant chapter.
• By chapter number — ask for ch07 to load that specific chapter file.
• Browse — ask "what chapters do you have?" to see the full index.

When you ask about a topic that's only briefly mentioned in Core Frameworks, I'll read the relevant chapter file before answering.

Core Frameworks & Mental Models

Spark in one paragraph

A unified engine for large-scale data processing. The driver orchestrates executors on a cluster; computation is expressed as a DAG of transformations (lazy) and actions (eager). Structured APIs (DataFrame, Dataset) are optimized by the Catalyst optimizer and compiled by Tungsten into compact JVM bytecode. The same engine powers batch (Spark SQL), streaming (Structured Streaming), ML (MLlib), and graph (GraphX) workloads.

Execution hierarchy (memorize for the exam)

Application → Job (per action) → Stage (split at every shuffle/exchange) → Task (one per partition, runs on one executor core).

Lazy evaluation + lineage

Transformations record a lineage DAG; actions trigger execution. Lineage = fault tolerance — Spark can rebuild lost partitions by replaying transformations on the source data.

Narrow vs Wide transformations

Narrow (filter, select, map, union): 1 input partition → 1 output partition, no shuffle. Wide (groupBy, orderBy, join, distinct, repartition): cross-partition data exchange → stage boundary.

Deployment modes (exam-critical)

Local mode is the only mode with driver+executors in one JVM.

Catalyst optimizer — 4 phases

1. Analysis — resolve names via Catalog → AST
2. Logical optimization — rule-based + cost-based (predicate pushdown, projection pruning, constant folding)
3. Physical planning — choose physical operators
4. Code generation — Tungsten whole-stage codegen → compact JVM bytecode

DataFrame API patterns

# Reader
df = (spark.read.format("csv").schema(schema)
      .option("header", "true").load(path))

# Writer (sample exam Q1)
df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

# Common modifications (exam Sec 3 staple)
df2 = (df.withColumn("state", col("division"))
         .drop("division")
         .withColumnRenamed("mName", "managerName"))

# Drop rows with ANY null
df.na.drop()              # all columns must be non-null
df.na.drop(subset="sqft") # check only one column

# Aggregations
df.groupBy("color").agg(count("*"), approx_count_distinct("user_id"), mean("amount"))

# Joins (inner is default)
a.join(broadcast(b), "key")     # force BHJ
a.join(b, "key", "leftOuter")   # explicit type

Save modes

overwrite · append · ignore · error/errorifexists (default)

Spark SQL: temp views

• Temp view: df.createOrReplaceTempView("v") — session-scoped
• Global temp view: df.createOrReplaceGlobalTempView("v") — cross-session, query as global_temp.v

Managed vs Unmanaged tables

• Managed: Spark owns data + metadata. DROP TABLE removes both.
• Unmanaged: Spark owns metadata only. DROP TABLE removes only metadata.

Tuning essentials (exam Sec 4)

• spark.sql.shuffle.partitions = 200 (default) — partition count after a wide transformation. Sample exam Q5 tests this exact fact.
• spark.sql.autoBroadcastJoinThreshold = 10 MB (default) — BHJ cutoff. -1 disables.
• spark.sql.files.maxPartitionBytes = 128 MB — file-read partition size.
• Executor memory: 300 MB reserved + 60% execution + 40% storage (defaults; can borrow between exec/storage).
• Cache vs Persist: cache() = MEMORY_AND_DISK (Python) / MEMORY_ONLY (Scala). persist(StorageLevel.X) for control. Cache is lazy — needs an action to materialize.
• AQE (spark.sql.adaptive.enabled): runtime replanning — coalesces partitions, swaps SMJ→BHJ, handles skew.
• Repartition (wide) to grow/balance; Coalesce (narrow) to shrink only.

Join strategies (auto-picked)

Spark UI

Port 4040 (driver host). Tabs: Jobs · Stages · Storage · Environment · Executors · SQL · (Structured Streaming in 3.0+). Debug signals: max task time ≫ median → data skew; high GC → memory starvation; high shuffle read blocked time → I/O bottleneck.

Structured Streaming — 5 steps

1. spark.readStream.format(...) → source
2. DataFrame transforms (mostly identical to batch)
3. .writeStream.format(...).outputMode(...) → sink + mode
4. .trigger(...).option("checkpointLocation", ...) → checkpoint required for exactly-once
5. .start() → returns StreamingQuery

Streaming output modes

• append (default) — only new rows; not allowed for aggregation without watermark
• complete — all rows of result table (small aggregates only)
• update — only changed rows since last trigger

Streaming triggers

default (ASAP) · processingTime="N seconds" · once (one batch, stop) · continuous (experimental ms latency)

Exactly-once requires all three

Replayable source + deterministic compute + idempotent sink

Watermark pattern (bounds state)

(df.withWatermark("eventTime", "10 minutes")
   .groupBy(window("eventTime", "5 minutes"), key)
   .agg(...))

withWatermark BEFORE groupBy, on the SAME time column. Limits state to (maxEventTime − delay).

Streaming dedup

df.dropDuplicates("col1", "col2")                                # state unbounded
df.withWatermark("eventTime", "1h").dropDuplicates("id", "eventTime")  # bounded

Stream–stream joins

• Inner: watermark + time range condition optional (state unbounded if omitted)
• Outer: watermark + time range mandatory

UDFs

Per-session, opaque to Catalyst. Prefer built-ins. For Python, use Pandas UDFs (@pandas_udf decorator, Arrow-backed, vectorized) — much faster than row-by-row.

Datasets vs DataFrames

Dataset[T] = typed; Scala/Java only. DataFrame = untyped (Dataset[Row] in Scala). Python and R get only DataFrame.

⚠ Gaps vs the 2026 exam

This book was published in 2020. The exam guide (Oct 2025) includes topics added to Spark after this book:

• Spark Connect (Spark 3.4+, 2023) — client/cluster decoupling. Exam Sec 6.
• Pandas API on Spark (pyspark.pandas, formerly Koalas, merged Spark 3.2). Exam Sec 7.
• AQE on by default in Spark 3.2+ (book shows it as opt-in in 3.0).

For these, supplement with official docs at spark.apache.org/docs/latest/.

Chapter Index

Topic Index

• Accumulators → ch07
• Actions → ch02
• AQE / Adaptive Query Execution → ch07, ch12
• Broadcast join → ch07, ch05
• Broadcast variables → ch07
• Bucketing → ch07
• Cache / Persist / StorageLevel → ch07
• Catalyst optimizer → ch03
• Catalog API → ch04
• Checkpoint location → ch08
• Coalesce vs Repartition → ch07
• Cluster manager → ch01
• DataFrame API → ch03, ch04, ch05
• DataFrameReader / Writer → ch04
• Dataset / Encoders → ch06
• Date/time functions (to_date, date_format, unix_timestamp, year/month/day…) → ch05
• Deployment modes → ch01
• Delta Lake → ch09
• Dropna / null handling → ch04, ch05
• Dynamic allocation → ch07
• Exactly-once → ch08
• Execution hierarchy (Job/Stage/Task) → ch02
• Explain / query plan → ch03, ch07
• JDBC → ch05
• Joins (inner, outer, broadcast, etc.) → ch05, ch07
• Lazy evaluation → ch02
• log4j / Driver / Executor logs → ch07
• MLlib / spark.ml → ch10
• MLflow → ch11
• Output modes (append/complete/update) → ch08
• Pandas UDFs → ch05, ch12
• Parquet → ch04
• Partition / partitioning → ch01, ch07
• partitionBy on write → ch04
• Repartition → ch07
• Save modes → ch04
• Schemas (DDL string, StructType) → ch03
• Shared variables (broadcast + accumulators) → ch07
• Shuffle / shuffle.partitions → ch07
• Spark Connect → not in book (see Gaps above)
• Spark SQL → ch04, ch05
• SparkSession → ch01, ch04
• Spark UI → ch07
• Stateful streaming / StateStore → ch08
• Stream–static / stream–stream joins → ch08
• Structured Streaming → ch08
• Temp view / global temp view → ch04
• Triggers (processingTime, once, continuous) → ch08
• Tungsten → ch03, ch06
• UDFs → ch05
• Watermark → ch08
• Window functions → ch05
• Whole-stage codegen → ch03

Supporting Files

• glossary.md — every key term with definition and chapter pointer
• patterns.md — concrete techniques: when, how, trade-offs
• cheatsheet.md — single-page exam reference + sample-Q answers

Scope & Limits

This skill covers the book "Learning Spark, 2nd Edition" content. For Spark features added after early 2020 (Spark Connect, Pandas API on Spark, newer AQE defaults), consult the official Apache Spark docs. For hands-on practice with PySpark notebooks, use Databricks Community Edition.