Why use a DynamicFrame when I know DataFrames?

DynamicFrames shine when the schema is inconsistent and you need to resolve conflicts, and they integrate with Glue features such as job bookmarks and catalog reads. For heavy transformations, convert to a DataFrame and use regular PySpark.

AWS Glue Module 4: Writing PySpark Scripts with GlueContext and DynamicFrames

Introduction

Script-based jobs give you full control. A Glue script uses standard PySpark plus Glue's own classes: GlueContext wraps the Spark session, and DynamicFrame is a flexible table type that tolerates messy schemas. This module writes a complete job and explains each part.

Prerequisites

Module 3 completed
Basic PySpark ideas (DataFrames, columns)

Step-by-Step Guide

Step 1: Start with the standard job structure

Every Glue Spark script imports the awsglue libraries, creates a SparkContext and GlueContext, initializes a Job, does its work, and ends with job.commit(). Calling commit is important: it records the job state, which job bookmarks use.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# ... your ETL code goes here ...

job.commit()

Step 2: Read a table from the Data Catalog

create_dynamic_frame.from_catalog reads a cataloged table into a DynamicFrame. The transformation_ctx argument is a unique label for this step that job bookmarks use, so always set it.

orders = glueContext.create_dynamic_frame.from_catalog(
    database="glue_lab",
    table_name="raw_orders",
    transformation_ctx="orders_source",
)
print(orders.count())
orders.printSchema()

Step 3: Understand DynamicFrames and DataFrames

A DynamicFrame is like a Spark DataFrame, but each record describes itself, so a column that holds both numbers and text in different rows can be represented and then resolved. Use DynamicFrames for ingestion and cleaning of messy data. Convert to a Spark DataFrame with toDF() when you want SQL, window functions, or the full DataFrame API, and back with DynamicFrame.fromDF.

from awsglue.dynamicframe import DynamicFrame

df = orders.toDF()
df.createOrReplaceTempView("orders")
daily = spark.sql("""
    SELECT order_date, count(*) AS orders, sum(amount) AS revenue
    FROM orders
    WHERE status <> 'CANCELLED'
    GROUP BY order_date
""")
daily_dyf = DynamicFrame.fromDF(daily, glueContext, "daily_dyf")

Step 4: Write partitioned Parquet to S3

write_dynamic_frame.from_options writes a DynamicFrame to S3 with a chosen format and options. Use partitionKeys to create Hive-style folders. Writing Parquet with partitions makes the output fast to query.

glueContext.write_dynamic_frame.from_options(
    frame=daily_dyf,
    connection_type="s3",
    format="glueparquet",
    connection_options={
        "path": "s3://my-glue-lab-123456/curated/daily_revenue/",
        "partitionKeys": ["order_date"],
    },
    format_options={"compression": "snappy"},
    transformation_ctx="daily_sink",
)

Step 5: Create and run the job from the CLI

Upload the script to S3, create a job that points at it, and start a run. Choose a small worker count and the newest Glue version. Then check the run state and read the logs in CloudWatch.

aws s3 cp daily_revenue.py s3://my-glue-lab-123456/scripts/daily_revenue.py

aws glue create-job --name daily-revenue-job --role GlueLabRole \
  --glue-version 5.0 --worker-type G.1X --number-of-workers 2 \
  --command Name=glueetl,ScriptLocation=s3://my-glue-lab-123456/scripts/daily_revenue.py,PythonVersion=3

aws glue start-job-run --job-name daily-revenue-job
aws glue get-job-runs --job-name daily-revenue-job --max-items 1

FAQs

Why use a DynamicFrame when I know DataFrames?: DynamicFrames shine when the schema is inconsistent and you need to resolve conflicts, and they integrate with Glue features such as job bookmarks and catalog reads. For heavy transformations, convert to a DataFrame and use regular PySpark.

Summary

A Glue script creates a GlueContext, reads DynamicFrames from the catalog with transformation_ctx, can convert to DataFrames for SQL, writes partitioned Parquet with write_dynamic_frame.from_options, and finishes with job.commit().