Apache Spark Knowledge Patch

Apache Spark Knowledge Patch

Use this skill when upgrading Spark applications, changing Spark SQL or PySpark code, implementing Data Source V2 connectors, operating Structured Streaming, using Spark Connect, or adjusting deployment and security configuration. Start with the migration hazards below, then open the reference whose topic matches the task.

Reference index

Upgrade triage

Before changing application code, identify the Spark line used by the driver, executors, clients, connectors, shuffle service, and History Server. Mixed components make changed defaults and bundled dependency versions especially important.

1. Audit removed runtime support and namespace changes.
2. Compare all explicitly set legacy switches with the changed defaults.
3. Re-run SQL tests with ANSI mode, parser, collation, view, map-key, and JDBC mapping behavior in mind.
4. Exercise checkpoints and state stores from a copy of production metadata.
5. Test Python schemas and UDF results through the Arrow path actually used in deployment.
6. Recompile connector and extension code against the target artifacts.
7. Review externally visible REST, UI, logging, and Kubernetes behavior.

Highest-impact breaking changes

Application and runtime boundaries

• Servlet-facing integrations must use the jakarta namespace after the 4.0 migration; javax servlet types no longer match Spark's interfaces.
• Mesos resource-manager support is removed.
• PySpark drops Python 3.8 in 4.0, Python 3.9 in 4.1, and official PyPy support in 4.2. Check pandas, NumPy, and PyArrow floors in the migration reference.
• Spark 4.2 can build and run on Java 25, while R 3.x is no longer supported.
• The old *.blacklist.* configuration aliases are ignored in 4.1; use their current names.

SQL behavior

• spark.sql.ansi.enabled defaults to true in 4.0. Invalid operations, overflow, parsing, and pandas API on Spark behavior may now fail rather than return permissive results.
• Bare CREATE TABLE follows spark.sql.sources.default instead of defaulting to Hive. Use spark.sql.legacy.createHiveTableByDefault=true only as a transition.
• File table reads use spark.sql.files.ignoreCorruptFiles and spark.sql.files.ignoreMissingFiles; AccessControlException and BlockMissingException remain hard failures.
• Floating-point map keys normalize -0.0 to 0.0; array set operations and collect_set later apply consistent Spark equality to negative zero and NaN.
• SQL's system.builtin and system.session namespaces can take precedence over persistent two-part names. Qualify the catalog or enable spark.sql.legacy.persistentCatalogFirst where collisions exist.
• Duplicate CTE names are rejected case-insensitively, while NATURAL JOIN key matching honors spark.sql.caseSensitive.

Python and Arrow

• Columnar PySpark exchange and regular Python UDFs and UDTFs use Arrow by default in 4.2. Disable the individual Arrow settings only while resolving a compatibility issue.
• Safe Arrow conversion defaults on in 4.1, turning integer overflow, truncation, and precision loss into errors.
• Binary SQL values map consistently to Python bytes in 4.1 rather than bytearray.
• NumPy ndarray DataFrame creation goes directly through Arrow in 4.2 and requires PyArrow; inferred schemas can differ from the old path.
• Spark Connect delays Python column validation in __getitem__ and later __getattr__. Use PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1 temporarily when callers require eager errors.

Streaming and state

• An AvailableNow query falls back to a single batch when any source lacks native support.
• AQE defaults on for stateless streaming in 4.1. Disable spark.sql.adaptive.streaming.stateless.enabled when validating a partitioning regression.
• A checkpoint containing commit or offset logs but no metadata file fails in 4.2. Restore the metadata or start from a new checkpoint; bypass verification only with an explicit duplicate-write risk assessment.
• Python simple stream readers must advance their end offset whenever they return a non-empty batch.
• Stable source and sink names allow streaming graph reordering without invalidating checkpoints; name nodes before evolving a durable query.

Changed-default quick reference

High-value SQL and connector additions

• SQL scripting is enabled and GA, with handlers, multiple-variable declarations, and later cursor support.
• Recursive CTEs, query parameter markers, QUALIFY, metric views, and NEAREST BY expand native query composition.
• VARIANT is GA with file-format support, shredding metadata, and colon field access. Spark 4.2 also adds native GEOMETRY and GEOGRAPHY types.
• Schema-evolving MERGE and INSERT, replacement inserts, and DataFrameWriterV2.MergeInto reduce connector-specific write logic.
• Data Source V2 gains CDC reads, multi-operation transactions, broader view DDL, metadata-only tables, partition-stat predicates, procedures, functions, constraints, and join pushdown.
• V1 inserts enforce NOT NULL; per-write options override session settings; dynamic partition overwrite with PathOutputCommitProtocol replaces the affected partitions.

High-value streaming and Python additions

• TransformWithState v2 supports multiple state variables, column families, timers, initial state, TTL, batch operation, and Avro-backed evolution.
• The state data source can inspect snapshots, change feeds, timers, operator metadata, and value, list, or map state.
• Stateless real-time mode begins with Scala and later gains a PySpark trigger.
• Python data sources support DSv2 writes, Arrow writers, filter pushdown, streaming writes, admission control, and AvailableNow.
• Arrow-native decorators and iterator APIs cover scalar, table, grouped, and grouped-aggregate UDF or UDTF workloads.
• PySpark can interrupt tagged operations, add artifacts, create DataFrames from PyArrow tables, and perform schema-evolving mergeInto.

High-value Connect, deployment, and security additions

• spark.api.mode selects Connect or classic behavior. Connect provides a lightweight Python client, Java compatibility, Scala Dataset parity, a JDBC driver, ML support, status APIs, and remote lifecycle controls.
• Declarative Pipelines manage graph ordering, parallelism, checkpoints, and retries; Auto CDC flows implement streaming SCD Type 1 upserts.
• Kubernetes gains Deployment API support, heterogeneous executor and PVC resize plugins, and a Stable resource-manager API.
• RPC supports AES-GCM and separate JKS key passwords.
• UI controls cover SNI host checks, Content-Security-Policy headers, and stack-trace suppression. JDBC URLs, job descriptions, Worker JSON, and temporary files receive stronger sensitive-output handling.

Validation focus

For an upgrade, validate behavior rather than only compilation:

• Compare query results for ANSI errors, null handling, NaN, negative zero, collations, grouping sets, natural joins, and JDBC type mappings.
• Inspect physical plans for AQE, pushdown, runtime filters, and partition behavior.
• Verify file writes with null constraints, dynamic overwrite, schema evolution, compression, and format annotations.
• Restart streaming queries from representative checkpoints and inspect state repair, source identity, and commit behavior.
• Test UDF and data-source boundaries with nulls, nested UDTs, multiple Arrow batches, nullable integer dtypes, and declared-schema mismatches.
• Confirm log processors, metrics consumers, REST exposure, network policy, History Server loading, redaction, and dependency overrides.