AWS Glue 5.1 Lands on Spark 3.5.6: What Data Teams Should Do

Any platform lead who has run a Sunday morning batch job knows the moment when a Spark version mismatch surfaces in a Slack thread nobody wants to read. AWS Glue has quietly moved its runtime twice in twelve months, and if your team is still pinning jobs to 4.0, you are now two major engine bumps behind. The current tutorial path from Amazon's serverless ETL front door lands directly on Glue 5.1, and the gap between where most teams are and where new projects should start has widened.

What Happened

As https reported in an August 21, 2026 walkthrough by Nadia Dubois, AWS Glue 5.1 was released on November 26, 2025 and now runs on Apache Spark 3.5.6, Python 3.11, and Scala 2.12.18. That release followed Glue 5.0, which shipped in late 2024 on Spark 3.5.4 and jumped the engine from the Spark 3.3.0 baseline that Glue 4.0 was stuck on. In practical terms, teams that skipped a cycle are looking at a two-minor-version Spark leap and a Python interpreter bump in the same migration.

Glue 5.0 brought more than Spark. It bundled Java 17, Hudi 0.15.0, Iceberg 1.7.1, and Delta Lake 3.3.0, and added support for Amazon SageMaker Unified Studio and SageMaker Lakehouse. Glue 5.1 sits on the same Python and Scala versions as 5.0 and, per AWS's own framing, adds performance and security enhancements on top of the Spark 3.5.6 bump.

The change with the biggest operational blast radius came in June 2025, when AWS added support for Glue 5.0 Spark jobs to read and write directly against AWS Lake Formation-registered tables, provided the job's IAM role has full table access. That is the kind of quiet feature drop that reshapes lakehouse access patterns without anyone posting a hot take about it.

Technical Anatomy

The Glue architecture has not fundamentally changed. You still have three pieces that matter: the Glue Data Catalog as a shared metadata store, Crawlers for schema discovery, and Jobs as the ETL compute. The Data Catalog remains the important part, because it is read by Athena, Redshift Spectrum, EMR, and Lake Formation. Catalog your data once, query it from four engines. That is the actual product.

What shifted underneath is the runtime. Going from Spark 3.3.0 in Glue 4.0 to Spark 3.5.6 in Glue 5.1 is not a patch bump. The Catalyst optimizer behaves differently, adaptive query execution defaults have moved, and Python 3.11 breaks anything pinned to older typing behavior or removed standard library modules. Teams I've worked with on Spark upgrades in similar version ranges consistently underestimate the UDF surface, especially where pandas UDFs and PyArrow versions interact. If you have Scala 2.12 jobs, you land softly, since Glue 5.x stays on Scala 2.12.18. Kotlin refugees and anyone eyeing Scala 2.13 do not get relief here.

The Lake Formation direct read/write change deserves its own paragraph. Before June 2025, running a Spark job against LF-governed tables meant working around the permission model or falling back to catalog-only reads. Now, if the job's IAM role has full table access, Glue 5.0 Spark can hit those tables natively. That is the piece that finally makes Glue viable as the primary compute for a Lake Formation-centric lakehouse, rather than a metadata-only participant. For teams standardizing on Iceberg via Lake Formation, this closes a real gap. For teams using Databricks-managed Delta on the same data, the story is less clean; the Databricks docs still assume you own the compute plane.

One detail worth flagging for cost-sensitive workloads: Glue Python Shell jobs run plain Python without spinning up a Spark cluster. Glue ETL compute is billed by the second on a DPU-hour basis, so a Python Shell job for a lightweight orchestration task can be an order of magnitude cheaper than a Spark job that spends most of its wall time on cluster startup. That is money left on the table by teams that default every job type to Spark.

Who Gets Burned

The teams facing the ugliest quarter are the ones running production Glue 4.0 jobs with heavy PySpark UDFs and no runtime pinning discipline. Two Spark minor versions and a Python 3.9 to 3.11 jump is not a lift-and-shift. Expect regression testing to eat real engineering time. On a ten-person data platform team, a two-engineer migration effort for a quarter is twenty percent of your capacity, and that is before anyone opens a ticket about a broken downstream Athena query.

iGaming and fintech shops running scheduled batch ETL against S3 data lakes are the obvious exposed category, especially where the same catalog feeds Athena dashboards for compliance reporting. The uncomfortable read: if you built your reporting stack on Glue 4.0 in 2023 and never touched it because it worked, you are now three years and two runtimes behind, and the deprecation clock is not paused for your convenience.

Lakehouse teams get the mixed news. If you have committed to Iceberg on Lake Formation, Glue 5.0 and 5.1 are the first Glue releases where the story hangs together end to end. If you have committed to Delta Lake with Databricks-managed compute and were using Glue as a cheap secondary engine, the value proposition is thinner. Delta Lake 3.3.0 is bundled in Glue 5.0, but the operational center of gravity for Delta remains elsewhere.

Teams running Snowflake as the warehouse and Glue only as the ingest tier are largely unaffected by the lakehouse changes, but should still care about the Python 3.11 bump. Any custom connector code that assumed 3.9 semantics needs a look. Snowflake's ingestion patterns are documented in the Snowflake docs if you are rethinking the boundary.

Playbook for Data Teams

My take: no new Glue project started in Q3 2026 should target anything older than 5.1. The runtime gap will only widen, and starting on 4.0 today is signing up for a migration you have not scoped.

Concrete actions for this week:

• Inventory your Glue jobs by version. If you cannot produce a spreadsheet of every job and its runtime by Friday, that is the first problem to solve. Tag jobs with the owning team while you are at it.
• Pin runtimes explicitly. Do not rely on defaults. The Glue versions release notes list exact bundled versions of Hadoop, Iceberg, Hudi, and Delta Lake per release. Pin against that matrix in your IaC.
• Audit UDFs for Python 3.11 compatibility. Anything using distutils, deprecated asyncio APIs, or older typing patterns needs attention before you flip a job to 5.1.
• Reclassify jobs by type. Any job that is really orchestration or lightweight munging should move to Python Shell, not Spark ETL. Glue is billed by DPU-hour, and Spark cluster startup on a 30-second workload is pure waste.
• Test Lake Formation integration in a sandbox. If you run governed tables, validate the June 2025 direct read/write path with a role that has full table access before you rewire production jobs.

For teams evaluating whether Glue is even the right tool, the honest answer depends on workload shape. If your transformations are SQL-first and your warehouse is the center of the world, dbt plus a warehouse engine will out-ergonomics Glue every time. Glue earns its DPU-hours when you have real Spark workloads, S3-native data, and a catalog you want shared across engines.

Key Takeaways

• AWS Glue 5.1 shipped November 26, 2025 on Spark 3.5.6, Python 3.11, and Scala 2.12.18. No new project should target an older runtime.
• The Spark 3.3.0 to 3.5.6 jump from Glue 4.0 to 5.1 is a real migration, not a version bump. Budget accordingly.
• June 2025's Lake Formation direct read/write support makes Glue viable as primary compute for LF-governed lakehouses, if the IAM role has full table access.
• Glue Python Shell jobs skip the Spark cluster entirely and are the right default for lightweight workloads billed by DPU-hour.
• The Data Catalog remains Glue's real moat: shared with Athena, Redshift Spectrum, EMR, and Lake Formation from a single crawl.