Here are some of the key takeaways from that experience.
One of the biggest surprises is how much networking still matters. EMR Serverless jobs typically run inside your VPC when accessing private resources, which means subnet configuration, routing, DNS, and endpoints must be correct. Misconfigured networking often leads to vague runtime failures that are harder to debug than simple infrastructure errors. Treat networking as a first-class design concern even when using serverless compute.
If your workloads interact heavily with S3 — which most EMR jobs do — a gateway VPC endpoint for S3 is almost always worth adding. Without it, traffic flows through the NAT gateway, increasing both latency and cost. This is especially noticeable in development environments where iterative testing can generate significant traffic. The endpoint simplifies connectivity and usually reduces operational friction.
Permissions tend to be the most time-consuming part of an EMR Serverless setup. The execution role needs access to multiple components simultaneously: EMR-managed artifacts, your own S3 buckets, CloudWatch logs, and networking resources like ENIs. Missing a single permission often results in runtime job failures rather than clear infrastructure errors. Starting with a functional policy and tightening it iteratively often works better than over-optimizing upfront.
While not strictly required, EMR Studio makes experimentation noticeably easier. Interactive notebooks, centralized access to logs, and a browser-based interface reduce the barrier for developers who are less familiar with EMR CLI workflows. For development environments focused on testing, prototyping, or onboarding new users, Studio is usually worth including.
Initial and maximum capacity settings for EMR Serverless directly impact both cost and performance. Conservative defaults combined with auto-start and auto-stop settings tend to work well for development environments. Too little capacity leads to throttled jobs, while overly generous limits can result in unnecessary cost spikes during testing.
Distributed data workloads rarely fail in obvious ways. Centralizing logs in CloudWatch from the beginning makes troubleshooting significantly easier. Retrofitting logging after problems appear is far more painful than enabling it upfront, even in non-production environments.
EMR Serverless simplifies infrastructure management, but it doesn’t eliminate architectural thinking. Networking, IAM design, cost awareness, and observability still matter. Treating it as “just another serverless service” often leads to unnecessary friction, while approaching it as a distributed data platform yields much better results.
These lessons informed the Terraform environment presented earlier in this series and continue to shape how I approach EMR-based workloads moving forward.
]]>The test suite is validating three essential capabilities of your EMR Serverless setup:
This is less about unit testing code and more about end-to-end platform verification.
Under terraform/projects/data/emr-serverless/tests/ you have three test scenarios:
Each scenario includes:
All tests follow the same runtime contract:
This pattern is excellent for a blog because readers can copy/paste and adapt quickly.
test_pi is your smoke test. It answers one question: “Can I run anything at all?”
Why it matters:
If this fails, there is no point running bigger jobs yet.
This is closer to a real dev workflow: submit a custom Python/Spark job with input data and process results.
Why it matters:
For most teams, this is the first “real confidence” test.
This scenario runs a larger benchmark-style workload.
Why it matters:
This is where platform tuning starts, not just platform validation.
This test setup is strong for a developer platform because it is:
If you publish this as a blog section, position it as:
That message makes the project feel production-minded, even in a dev/test environment.
]]>The goal here is not “deploy infrastructure for the sake of it.” The goal is to remove friction from testing distributed data workloads.
The diagram below summarizes how the main pieces fit together: inputs (variables) drive a VPC with public and private subnets, EMR Serverless and EMR Studio in the private tier, an S3 gateway endpoint for in-VPC access to the bucket, and regional resources (S3, CloudWatch) plus IAM roles wired to the outputs you use from scripts and CI.
If you’ve ever tried to test EMR seriously, you probably noticed something: it’s not trivial to spin up a safe, reproducible dev environment. Permissions, networking, logging, data access, runtime dependencies — everything needs to line up, and when one piece is missing, debugging quickly becomes painful.
This stack gives developers a controlled space to:
Everything is split into logical Terraform files so the environment remains understandable and maintainable rather than becoming a monolithic IaC blob.
This table summarizes the Terraform resources that make up the EMR Serverless development environment: networking, compute, security, IAM, storage, and observability.
| Layer | Resource / module | Purpose |
|---|---|---|
| Networking | module.vpc | VPC, 2 public + 2 private subnets, 1 NAT, IGW |
| Networking | aws_vpc_endpoint.s3 | S3 Gateway endpoint on private route tables |
| Compute | aws_emrserverless_application.main | Spark app (driver + executors in private subnets) |
| Compute | aws_emr_studio.main | EMR Studio (notebooks, in private subnets) |
| Security | aws_security_group.emr_serverless | Single SG for app + studio (egress only) |
| IAM | aws_iam_role.emr_serverless_execution | Role + inline policy (S3, CW, EC2 ENI in VPC) |
| IAM | aws_iam_role.emr_studio | Role + Editors, S3Full, EMRFull |
| Storage | aws_s3_bucket.emr | One bucket: logs, libraries, studio, tests |
| Observability | aws_cloudwatch_log_group.emr | Log group for EMR Serverless |
This file doesn’t create infrastructure. It teaches Terraform about the environment it’s operating in and builds IAM policy documents dynamically.
These data sources remove hardcoded assumptions:
That might sound minor, but this is what makes infrastructure portable instead of brittle.
This is where most EMR deployments either become secure… or messy.
We generate:
Using aws_iam_policy_document keeps policies readable, composable, and version-controlled. It produces JSON only; actual IAM roles attach it later.
This file turns design into running infrastructure.
The VPC module creates:
An S3 VPC endpoint is added intentionally. Without it, Spark jobs in private subnets would route S3 traffic through NAT, increasing cost and latency. This small addition usually pays for itself quickly.
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "5.8.1"
name = "${var.project_name}-vpc"
cidr = "10.100.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 2)
private_subnets = ["10.100.1.0/24", "10.100.2.0/24"]
public_subnets = ["10.100.101.0/24", "10.100.102.0/24"]
enable_nat_gateway = true
single_nat_gateway = true
enable_dns_hostnames = true
enable_dns_support = true
tags = merge(var.tags, { Name = "${var.project_name}-vpc" })
}
resource "aws_vpc_endpoint" "s3" {
vpc_id = module.vpc.vpc_id
service_name = "com.amazonaws.${data.aws_region.current.region}.s3"
vpc_endpoint_type = "Gateway"
route_table_ids = module.vpc.private_route_table_ids
tags = merge(var.tags, { Name = "${var.project_name}-s3-ep" })
}
The S3 bucket becomes the central artifact store:
Public access is blocked, versioning protects against accidental overwrites, and encryption is enforced by default. These aren’t luxury settings; they’re baseline operational hygiene.
resource "aws_s3_bucket" "emr" {
bucket = "${var.project_name}-bucket-${data.aws_caller_identity.current.account_id}"
force_destroy = true
tags = merge(var.tags, { Name = "${var.project_name}-bucket" })
}
resource "aws_s3_bucket_public_access_block" "emr" {
bucket = aws_s3_bucket.emr.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
A dedicated CloudWatch log group captures driver and executor logs. When something fails in distributed processing, logs are often the only way to understand what happened. Centralizing logs from day one avoids painful retrofits later.
resource "aws_cloudwatch_log_group" "emr" {
name = "/aws/emr-serverless/${var.project_name}"
retention_in_days = 7
tags = merge(var.tags, { Name = "${var.project_name}-log-group" })
}
Two roles matter here:
Security groups remain intentionally simple: outbound allowed, inbound restricted. EMR Serverless relies heavily on AWS-managed service communication, so overly strict network rules often cause subtle failures that are difficult to diagnose.
resource "aws_iam_role" "emr_serverless_execution" {
name = "${var.project_name}-execution-role"
assume_role_policy = data.aws_iam_policy_document.emr_serverless_assume.json
tags = merge(var.tags, { Name = "${var.project_name}-execution-role" })
}
resource "aws_iam_role_policy" "emr_serverless_execution" {
name = "${var.project_name}-execution-policy"
role = aws_iam_role.emr_serverless_execution.id
policy = data.aws_iam_policy_document.emr_serverless_execution.json
}
resource "aws_iam_role" "emr_studio" {
name = "${var.project_name}-studio-role"
assume_role_policy = data.aws_iam_policy_document.emr_studio_assume.json
tags = merge(var.tags, { Name = "${var.project_name}-studio-role" })
}
resource "aws_iam_role_policy_attachment" "emr_studio_editors" {
role = aws_iam_role.emr_studio.name
policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/service-role/AmazonElasticMapReduceEditorsRole"
}
resource "aws_iam_role_policy_attachment" "emr_studio_s3" {
role = aws_iam_role.emr_studio.name
policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonS3FullAccess"
}
resource "aws_iam_role_policy_attachment" "emr_studio_emr" {
role = aws_iam_role.emr_studio.name
policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonElasticMapReduceFullAccess"
}
This is where the environment becomes usable:
Without Studio, EMR experimentation tends to become CLI-heavy and slower for most teams.
resource "aws_emrserverless_application" "main" {
name = "${var.project_name}-app"
release_label = var.emr_release_label
type = var.emr_serverless.application_type
architecture = var.emr_serverless.architecture
auto_start_configuration {
enabled = true
}
auto_stop_configuration {
enabled = true
idle_timeout_minutes = var.emr_serverless.auto_stop_idle_minutes
}
network_configuration {
subnet_ids = module.vpc.private_subnets
security_group_ids = [aws_security_group.emr_serverless.id]
}
initial_capacity {
initial_capacity_type = "Driver"
initial_capacity_config {
worker_count = var.emr_serverless.driver.worker_count
worker_configuration {
cpu = var.emr_serverless.driver.cpu
memory = var.emr_serverless.driver.memory
disk = var.emr_serverless.driver.disk
}
}
}
initial_capacity {
initial_capacity_type = "Executor"
initial_capacity_config {
worker_count = var.emr_serverless.executor.worker_count
worker_configuration {
cpu = var.emr_serverless.executor.cpu
memory = var.emr_serverless.executor.memory
disk = var.emr_serverless.executor.disk
}
}
}
maximum_capacity {
cpu = var.emr_serverless.maximum_capacity.cpu
memory = var.emr_serverless.maximum_capacity.memory
disk = var.emr_serverless.maximum_capacity.disk
}
tags = var.tags
}
resource "aws_emr_studio" "main" {
name = "${var.project_name}-studio"
auth_mode = "IAM"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
service_role = aws_iam_role.emr_studio.arn
workspace_security_group_id = aws_security_group.emr_serverless.id
engine_security_group_id = aws_security_group.emr_serverless.id
default_s3_location = "s3://${aws_s3_bucket.emr.id}/studio/"
tags = var.tags
}
This file abstracts environment-specific settings:
Separating configuration from infrastructure code keeps the module reusable across dev, staging, or experimental environments without rewriting resources.
Outputs expose what engineers actually need after deployment:
These values typically feed CI pipelines, test scripts, or job submission tooling.
These files define concrete environment values. Think of them as profiles:
This separation makes spinning up multiple environments predictable.
This environment isn’t intended to be production-ready out of the box. It’s designed to make experimentation with EMR Serverless practical: reproducible infrastructure, sensible defaults, and enough observability to understand what your workloads are actually doing.
In the next post, I’ll move from infrastructure into running real Spark workloads and what that experience looks like in practice.
Repository Reference
The complete Terraform configuration used in this article is available in GitHub.
The repository includes the full infrastructure code, example configuration files, diagrams, and deployment guidance so you can reproduce the environment, experiment safely, or adapt it to your own workloads.
Feel free to explore the code, open issues, or reuse the setup as a starting point for building your own EMR Serverless development environments.
]]>Before jumping into the theory, I want to share something from real-world experience. EMR is powerful, but getting started with it isn’t always straightforward. For many developers — even experienced ones — testing EMR workloads locally or in a safe dev environment can be challenging. The ecosystem is broad, the configurations can be nuanced, and understanding how all the moving parts connect sometimes takes longer than expected.
From an infrastructure and DevOps perspective, I’ve found that creating a reproducible environment with Infrastructure as Code makes a huge difference. In my case, Terraform allowed me to build a consistent, cost-controlled development setup where experimentation becomes practical instead of painful. In the second part of this series, I’ll walk through how I approached building that environment so you can test, learn, and iterate without unnecessary friction.
Amazon EMR (Elastic MapReduce) is a managed big data platform from AWS designed to process and analyze large datasets at scale. It simplifies running distributed data processing frameworks such as Apache Spark, Hadoop, Hive, Presto, Flink, and others without requiring you to manually provision, configure, or maintain complex cluster infrastructure.
Instead of building and operating your own big data environment, EMR provides managed compute resources, integrations with AWS storage services like Amazon S3, automated scaling capabilities, and built-in monitoring. This allows teams to focus on extracting insights from data rather than managing infrastructure.
EMR can run in multiple deployment models depending on operational needs:
This flexibility makes EMR suitable for a wide range of data processing scenarios, from exploratory analytics to production-grade pipelines.
One of the most common uses of EMR is processing massive datasets for Extract, Transform, Load (ETL) workflows. Spark or Hive jobs can clean, transform, and aggregate data stored in S3, preparing it for analytics platforms, machine learning workflows, or downstream applications.
Typical examples include:
EMR supports interactive SQL engines such as Presto or Spark SQL, enabling data scientists and analysts to query large datasets directly without moving data into traditional databases.
This is often used for:
Machine learning workflows often require large-scale preprocessing before training models. EMR is frequently used to:
Because it integrates easily with services like SageMaker, EMR often serves as the data preparation layer in ML pipelines.
With frameworks such as Apache Flink or Spark Streaming, EMR can handle near-real-time data processing scenarios, including:
This allows organizations to combine batch and streaming workloads in the same platform.
Many organizations use EMR as the compute layer on top of an Amazon S3 data lake. This approach separates storage from compute, providing flexibility, cost efficiency, and scalability.
Typical architecture patterns include:
In short, AWS EMR is designed to simplify large-scale data processing while offering multiple deployment options to balance control, cost, and operational overhead. It is particularly valuable when workloads involve large datasets, distributed processing, and integration with modern cloud data architectures.
]]>I’ve been working in DevOps, cloud infrastructure, and automation for several years, building platforms, migrating workloads, troubleshooting production issues, and constantly learning how to do things better. This blog is simply a place to capture that journey — the wins, the mistakes, the lessons, and the ideas worth sharing.
Most of my day-to-day work revolves around AWS, Terraform, Docker, and Kubernetes, along with automation, platform engineering, CI/CD, and reliability practices. I’m particularly interested in building infrastructure that is practical, scalable, and maintainable — not just technically correct, but operationally sane. Recently, I’ve also started exploring how AI intersects with DevOps and cloud engineering, which is quickly becoming an exciting new layer in the ecosystem.
This isn’t meant to be a polished tutorial hub or marketing-style content. Think of it more as an engineering notebook: real experiences, architecture decisions, tooling experiments, lessons learned the hard way, and occasional opinions about where infrastructure and cloud technology are heading.
Outside of tech, life stays balanced thanks to video games, time with my family, and dogs — things that remind me that building cool technology should still be enjoyable and meaningful.
If anything you read here helps you solve a problem, learn something new, or see infrastructure from a slightly different angle, then the blog is doing exactly what it’s meant to do.
]]>