ACK Controller for EMR on EKS

Introduction

In this post, we will learn to build EMR on EKS Spark workloads by using AWS Controllers for Kubernetes (ACK). We will also build a end-to-end observability for Spark workloads by leveraging Amazon Managed Service for Prometheus to collect and store the metrics generated by Spark Applications and then use Amazon Managed Grafana to build dashboards for monitoring use cases.

Deploying the Solution

👈

Setup Amazon Managed Grafana with SSO

Currently, this step is manual. Please follow the steps in this blog to create Amazon Managed Grafana with SSO enabled in your account. You can visualize the Spark jobs runs and metrics using Amazon Managed Prometheus and Amazon Managed Grafana.

Execute Sample Spark job - EMR Virtual Cluster

👈

Cleanup

👈

caution

To avoid unwanted charges to your AWS account, delete all the AWS resources created during this deployment