My bookmarked job processed nothing. Why?

Check that there truly is new data, that the source has a transformation_ctx, and that the earlier run called job.commit(). If you changed the script or source, reset the bookmark to reprocess.

AWS Glue Module 6: Partitions, Pushdown Predicates, and Job Bookmarks

Introduction

Reading and rewriting all your data on every run is slow and costly. Two Glue features avoid it: pushdown predicates read only matching partitions, and job bookmarks remember what a job already processed so the next run handles only new data. This module uses both.

Prerequisites

Module 5 completed
Partitioned data in S3 (for example ingest_date=2026-09-25/)

Step-by-Step Guide

Step 1: Use a pushdown predicate to read only some partitions

A pushdown predicate filters partitions in the catalog before any data is read, so Spark never scans the other folders. The predicate is a SQL-like expression on partition columns. This can cut runtime and cost dramatically on large tables.

orders = glueContext.create_dynamic_frame.from_catalog(
    database="glue_lab",
    table_name="raw_orders_partitioned",
    push_down_predicate="ingest_date >= '2026-09-20' and ingest_date <= '2026-09-25'",
    transformation_ctx="orders_source",
)

Step 2: Turn on job bookmarks

A job bookmark stores the state of the data already processed, so the next run of the same job reads only new files (for S3 sources) or new rows (for supported JDBC sources with a key). Enable it with a job parameter, give every source and sink a transformation_ctx, and call job.commit() at the end. Bookmarks only work if those three things are in place.

aws glue update-job --job-name daily-revenue-job --job-update '{
  "Role": "GlueLabRole",
  "Command": {"Name": "glueetl", "ScriptLocation": "s3://my-glue-lab-123456/scripts/daily_revenue.py", "PythonVersion": "3"},
  "DefaultArguments": {"--job-bookmark-option": "job-bookmark-enable"}
}'

Step 3: Know the bookmark options and limits

The options are enable (track and use state), disable (process everything every time), and pause (use existing state without updating it). Bookmarks track files by modification, so rewriting old files makes them look new. To reprocess from scratch, reset the bookmark. They suit append-style data, not data that is updated in place.

aws glue reset-job-bookmark --job-name daily-revenue-job

Step 4: Write partitioned output and add partitions to the catalog

Write results partitioned by a column you filter on. If the output is registered in the catalog, new partitions must be added: have the job update the catalog by enabling partition updates in the sink options, run a crawler, or add partitions with an ALTER TABLE statement.

sink = glueContext.getSink(
    path="s3://my-glue-lab-123456/curated/orders/",
    connection_type="s3",
    updateBehavior="UPDATE_IN_DATABASE",
    partitionKeys=["order_date"],
    enableUpdateCatalog=True,
    transformation_ctx="orders_sink",
)
sink.setCatalogInfo(catalogDatabase="glue_lab", catalogTableName="curated_orders")
sink.setFormat("glueparquet")
sink.writeFrame(shipped)

Step 5: Avoid small files

Many tiny output files slow later queries. Control file counts by repartitioning or coalescing before the write, or let Glue group small input files with the groupFiles option when reading many small files. Aim for output files in the range of hundreds of megabytes, and check the guidance for your query engine.

shipped_df = shipped.toDF().repartition(4)    # roughly 4 output files per run
shipped = DynamicFrame.fromDF(shipped_df, glueContext, "shipped")

# when reading many small input files
src = glueContext.create_dynamic_frame.from_options(
    "s3",
    {"paths": ["s3://my-glue-lab-123456/raw/orders/"], "recurse": True,
     "groupFiles": "inPartition", "groupSize": "134217728"},
    format="json",
    transformation_ctx="small_files_source",
)

FAQs

My bookmarked job processed nothing. Why?: Check that there truly is new data, that the source has a transformation_ctx, and that the earlier run called job.commit(). If you changed the script or source, reset the bookmark to reprocess.

Summary

Push partition filters down to read less data, enable job bookmarks with transformation_ctx and job.commit() to process only new data, write partitioned output and keep the catalog updated, and control file sizes.