Earlier in this series I built an EMR Serverless environment with Terraform and described the test suite that validates it. This post covers the smallest of those tests, the one I run most: Spark Pi.

It answers a single question in a couple of minutes — can this environment run anything at all? When a deployment is broken it is almost never broken in an interesting way. It is a missing IAM permission, a subnet with no route, or a log group the job cannot write to.

Why Spark Pi

The math is irrelevant. What matters is everything that must work before the first point is thrown: the application starts, the execution role is assumable, workers get network interfaces in your private subnets, the runtime pulls its own artifacts, and logs arrive somewhere you can read them.

Pi exercises that whole path, and it ships inside the EMR runtime image, so there is nothing to upload first:

/usr/lib/spark/examples/jars/spark-examples.jar

That path is local to the worker, not S3 — which removes your bucket, your packaging and your code from the equation. If Pi fails, the problem is the platform. If Pi passes and your real job fails, the problem is your job. That separation is worth a lot when you are debugging at 6pm.

Start from the Terraform outputs

The stack exposes everything the test needs, so nothing is hardcoded:

export APPLICATION_ID=$(terraform output -raw application_id)
export EXECUTION_ROLE_ARN=$(terraform output -raw execution_role_arn)
export BUCKET_NAME=$(terraform output -raw bucket_name)
export LOG_GROUP_NAME=$(terraform output -raw log_group_name)

These variables are the contract between the infrastructure and the tests. Because the scripts read them instead of embedding IDs, the same test runs unchanged against any environment built from the stack.

Submitting the job

export JOB_RUN_ID=$(aws emr-serverless start-job-run \
  --application-id "$APPLICATION_ID" \
  --execution-role-arn "$EXECUTION_ROLE_ARN" \
  --name "smoke-test-pi" \
  --job-driver '{
    "sparkSubmit": {
      "entryPoint": "/usr/lib/spark/examples/jars/spark-examples.jar",
      "entryPointArguments": ["1"],
      "sparkSubmitParameters": "--class org.apache.spark.examples.SparkPi --conf spark.executor.cores=1 --conf spark.executor.memory=2g --conf spark.driver.cores=1 --conf spark.driver.memory=2g --conf spark.executor.instances=1"
    }
  }' \
  --query 'jobRunId' --output text)

Three notes. entryPointArguments: ["1"] is the partition count — one is enough, and raising it only makes the job do more work without testing anything new. --class is required for JAR jobs and belongs in sparkSubmitParameters, not in the entry point. And the capacity is deliberately small: AWS documents this example with four cores and 20 GB of executor memory, which is right for a benchmark and wasteful for a smoke test.

Sending logs where you can find them

CloudWatch logging is off by default for EMR Serverless job runs — provisioning the log group in Terraform does not make jobs write to it. You ask for it per submission:

--configuration-overrides '{
  "monitoringConfiguration": {
    "cloudWatchLoggingConfiguration": {
      "enabled": true,
      "logGroupName": "'"$LOG_GROUP_NAME"'",
      "logTypes": { "SPARK_DRIVER": ["stdout", "stderr"] }
    },
    "s3MonitoringConfiguration": {
      "logUri": "s3://'"$BUCKET_NAME"'/logs/"
    }
  }
}'

CloudWatch gives you live tail while the job runs, which is how you watch something that is hanging rather than failing. S3 gets the full container logs afterwards, executors included — which matters the moment you adapt this script for a real job, because EMR Serverless publishes only driver stdout and stderr to CloudWatch by default. For Pi the driver is enough; for anything else, the stack trace is usually in an executor.

Watching the run

aws emr-serverless get-job-run \
  --application-id "$APPLICATION_ID" \
  --job-run-id "$JOB_RUN_ID" \
  --query 'jobRun.{state:state,details:stateDetails}'

A healthy run moves SUBMITTED → PENDING → SCHEDULED → RUNNING → SUCCESS. On a cold application the first couple of minutes pass before RUNNING even starts, because the application has to wake up and provision workers; AWS puts a typical run at three to five minutes end to end.

The driver then prints one line to stdout:

Pi is roughly 3.1417357086785433

In CloudWatch it lands under /applications/<applicationId>/jobs/<jobId>/SPARK_DRIVER/, with the same prefix inside your S3 log URI. The value differs every run — that is the algorithm, not a bug.

What the failures mean

What you see Where to look first
Stuck in SCHEDULED, never reaches RUNNING Networking: subnets, route tables, security group
FAILED before any Spark output Role trust policy, or no access to EMR runtime artifacts
Finishes, but no logs Logging not enabled, or missing logs:CreateLogStream / logs:PutLogEvents
AccessDenied on S3 Execution policy scope — check the bucket and /* ARNs
Far slower than expected Cold start plus initial_capacity and the idle timeout

An EMR Serverless failure is rarely a Spark problem. It is an IAM or networking problem wearing a Spark error message, which was the lesson of the previous post, and Pi is the cheapest way to surface it.

Wrap the sequence in a script that exits non-zero when the state is not SUCCESS, and it runs in CI right after terraform apply. That turns “the environment is probably fine” into something the pipeline asserts — the difference between a test and a ritual.

Four posts went into understanding EMR, building the environment, structuring the tests and collecting the lessons. This one estimates π badly and tells you nothing about your data pipeline. It just tells you the platform works, before you waste an afternoon debugging a job that was never going to start.

Repository Reference

The Terraform configuration is on GitHub, including the networking, IAM, storage and observability setup this test depends on.

References