Once your Terraform stack is up, the test layer is what proves the environment is actually usable for day-to-day development. In this project, tests are intentionally script-driven: simple, repeatable, and close to how engineers really submit Spark jobs.

What the tests are validating

The test suite is validating three essential capabilities of your EMR Serverless setup:

  • You can submit jobs to the EMR Serverless application successfully.
  • The execution role has the right permissions for S3 and CloudWatch.
  • The single project S3 bucket works as the artifact/input/output location.
  • Logs are emitted to CloudWatch and can be inspected for debugging.

This is less about unit testing code and more about end-to-end platform verification.

Test structure

Under terraform/projects/data/emr-serverless/tests/ you have three test scenarios:

  • test_pi/: a minimal Spark Pi validation.
  • test_food/: a data-processing job over a sample CSV dataset.
  • benchmark/: a heavier benchmark-style run using the Spark benchmark JAR.

Each scenario includes:

  • a markdown guide (.md) describing steps,
  • a runnable shell script (submit_job*.sh) that submits the job,
  • and, where needed, sample data or job code.

Shared test pattern

All tests follow the same runtime contract:

  1. Read Terraform outputs from the deployed stack:
    • application_id
    • execution_role_arn
    • bucket_name
    • log_group_name
    • region
  2. Export those values as environment variables.
  3. Run a submission script (submit_job_pi.sh, submit_job_health.sh, submit_job.sh).
  4. Inspect job run status with aws emr-serverless get-job-run.

This pattern is excellent for a blog because readers can copy/paste and adapt quickly.

Test 1: Spark Pi (test_pi)

test_pi is your smoke test. It answers one question: “Can I run anything at all?”

Why it matters:

  • fastest feedback loop,
  • verifies app availability, role trust, and basic Spark runtime path,
  • ideal as the first check after terraform apply.

If this fails, there is no point running bigger jobs yet.

Test 2: Food dataset job (test_food)

This is closer to a real dev workflow: submit a custom Python/Spark job with input data and process results.

Why it matters:

  • validates project bucket read/write behavior,
  • validates job packaging and submission arguments,
  • validates end-to-end data path (input -> processing -> output/logs).

For most teams, this is the first “real confidence” test.

Test 3: Benchmark run (benchmark)

This scenario runs a larger benchmark-style workload.

Why it matters:

  • validates capacity and scaling behavior under heavier load,
  • helps tune initial_capacity, maximum_capacity, and idle timeout,
  • surfaces bottlenecks in startup time, artifact loading, and logging behavior.

This is where platform tuning starts, not just platform validation.

Why this test design works well

This test setup is strong for a developer platform because it is:

  • Deterministic: all scripts consume Terraform outputs, not hardcoded IDs.
  • Portable: same scripts work across environments if outputs are exported.
  • Operational: uses the exact AWS CLI path teams use in CI/CD and troubleshooting.
  • Incremental: smoke test -> functional test -> performance-oriented test.

Suggested blog framing

If you publish this as a blog section, position it as:

  • “Deploy infrastructure with Terraform.”
  • “Validate with three progressive test levels.”
  • “Use outputs + scripts as the stable contract between infra and workloads.”

That message makes the project feel production-minded, even in a dev/test environment.