What are the benefits of Amazon EMR? Drawbacks of AWS EMR

Advantages of utilising Amazon EMR

Amazon EMR offers numerous advantages. These include AWS’s flexibility and the financial advantages when compared to developing your own on-premises resources.

Cost savings

The cost of Amazon EMR is determined on the type of instance, the quantity of Amazon EC2 instances you deploy, and the region in which your cluster is launched. Low prices are available with on-demand pricing, but you can save even more money by buying Reserved Instances or Spot Instances. Spot instances can save a lot of money; in certain situations, they can be as little as a tenth of on-demand cost.

Note

There are extra fees for using Amazon S3, Amazon Kinesis, or DynamoDB with your EMR cluster, and those fees are charged independently of your Amazon EMR usage.

Note

Setting up VPC endpoints for Amazon S3 is advised when establishing an Amazon EMR cluster in a private subnet. Because the connection between your EMR cluster and S3 will not remain within your VPC, you will be charged extra for NAT gateways related to S3 traffic if your EMR cluster is located in a private subnet without VPC endpoints for Amazon S3.

For additional facts and pricing choices, view Amazon EMR pricing.

AWS integration

In order to give your cluster networking, storage, security, and other features, Amazon EMR interfaces with other AWS services. There are numerous instances of this integration in the list that follows:

For the instances that make up the cluster’s nodes, use Amazon EC2.
Use Amazon Virtual Private Cloud (Amazon VPC) to set up the virtual network where your instances are launched.
storing input and output data on Amazon S3
Configure alarms and track cluster performance with Amazon CloudWatch.
Permissions configuration with AWS Identity and Access Management (IAM)
To audit service requests, use AWS CloudTrail.
AWS Data Pipeline for cluster scheduling and startup
Data may be found, categorised, and secured in an Amazon S3 data lake using AWS Lake Formation.

The deployment

The EC2 instances that make up your EMR cluster carry out the tasks that you assign to them. Amazon EMR sets up the instances with your selected apps, such Spark or Apache Hadoop, when you first launch your cluster. Streaming data, low-latency queries, batch processing, or big data storage pick the instance size and type that best fits your cluster’s processing requirements.

There are numerous ways to set up software on your cluster with Amazon EMR. An Amazon EMR release, for instance, can be installed with a selected collection of apps, which may include programs like Hive, Pig, or Spark as well as flexible frameworks like Hadoop. Installing one of the various MapR distributions is another option. You can also manually install software on your cluster using the yum package manager or from the source code because Amazon EMR runs on Amazon Linux.

Both flexibility and scalability

As your computing requirements change, you can scale your cluster up or down with Amazon EMR. Resizing your cluster allows you to add instances during periods of high workload and remove instances to reduce expenses when those periods end.

The ability to run several instance groups is another feature that Amazon EMR offers. This allows you to use Spot Instances in one group to finish your tasks more quickly and affordably, and On-Demand Instances in another group for guaranteed processing power. In order to benefit from a Spot Instance type’s better cost than another, you can also mix several instance kinds.

The ability to use several file systems for your input, output, and intermediate data is another feature that Amazon EMR offers. To process data that you do not need to store after your cluster’s lifecycle, you may, for instance, use the Hadoop Distributed File System (HDFS), which operates on the primary and core nodes of your cluster.

To decouple computation and storage and to store data outside of your cluster’s lifespan, you may decide to use Amazon S3 as a data layer for applications running on your cluster using the EMR File System (EMRFS). EMRFS offers the extra advantage of enabling you to autonomously scale up or down to meet your storage and computation requirements. You can use Amazon S3 to scale your storage demands and resize your cluster to accommodate your growing computing needs.

Reliability

Your cluster’s nodes are monitored by Amazon EMR, which automatically shuts down and replaces instances as necessary.

You can choose whether your cluster is terminated automatically or manually with the configuration options that Amazon EMR offers. If you set up your cluster to end automatically, it does so when all the procedures are finished. It is known as a transitory cluster. When processing is finished, you can set up the cluster to continue operating, giving you the option to manually stop it when you’re done using it. You can also build a cluster, use the installed apps directly, and then manually end the cluster when you’re done with it. We call the clusters in these examples “long-running clusters.”

To stop instances in your cluster from being terminated because of processing faults or problems, you may also set up termination protection. You can retrieve data from instances prior to termination when termination protection is activated. Whether you activate your cluster using the console, CLI, or API affects the default settings for these features.

Security

Amazon EMR helps you protect your data and clusters by utilising features like Amazon EC2 key pairs and other AWS services like IAM and Amazon VPC.

IAM

To manage permissions, Amazon EMR interfaces with IAM. IAM policies set permissions for people or groups. Policies grant users and groups access to resources and activities.

The EC2 instance profile is used for the instances, while IAM roles are used for the Amazon EMR service itself. These roles give the service and instances the authority to access other AWS services on your behalf. Both the Amazon EMR service and the EC2 instance profile have default roles. AWS managed policies, which are generated automatically when you launch an EMR cluster from the console and select default permissions, are used by the default roles. Additionally, you can use the AWS CLI to generate the default IAM roles. If you would rather manage the rights outside of AWS, you can create custom roles for the service and instance profile.

Groups for security

To regulate incoming and outgoing traffic to your EC2 instances, Amazon EMR uses security groups. When your cluster is launched, Amazon EMR employs a security group for your primary instance and another for your core/task instances to share. To guarantee communication amongst the cluster’s instances, Amazon EMR sets up the security group rules. More sophisticated restrictions can be used by configuring extra security groups and allocating them to your primary and core/task instances.

Encryption

The optional server-side and client-side encryption with EMRFS that Amazon EMR offers helps safeguard the data you save on Amazon S3. Amazon S3 secures your data once you submit it using server-side encryption.

When using client-side encryption, the EMRFS client on your EMR cluster handles both the encryption and decryption. Your key management system or AWS KMS can handle client-side encryption root keys.

Amazon VPC

Clusters can be launched in an Amazon VPC virtual private cloud (VPC) using Amazon EMR. An isolated virtual network in AWS, a VPC gives you control over more complex network settings and access features.

AWS CloudTrail

CloudTrail and Amazon EMR capture AWS account requests. This data lets you see who is accessing your cluster, when, and from what IP address.

Key pairs for Amazon EC2

Establishing a secure connection between the primary node and your distant computer allows you to monitor and communicate with your cluster. This connection can authenticate with SSH or Kerberos. You will need an Amazon EC2 key pair if you utilise SSH.

Monitoring

Cluster issues, like as faults or failures, can be debugged using the log files and Amazon EMR management interfaces. Log files can be archived on Amazon S3 using Amazon EMR, allowing you to save records and address problems even after your cluster has ended. Additionally, the Amazon EMR console offers an extra debugging tool for perusing the log files according to tasks, jobs, and steps.

In order to monitor cluster and job performance indicators, Amazon EMR interfaces with CloudWatch. Alarms can be set up according to a number of parameters, including the cluster’s idle status and storage usage %.

Management interfaces

There are several ways to access Amazon EMR:

The console is a graphical user interface that may be used to launch and manage clusters. It allows you to view the details of existing clusters, troubleshoot clusters, terminate clusters, and describe the details of clusters to launch via online forms. No programming experience is necessary to begin using Amazon EMR; the console is the simplest method. Online access to the console can be found at https://console.aws.amazon.com/elasticmapreduce/home.

The AWS Command Line Interface (AWS CLI) is a client program that you install on your computer to establish a connection to Amazon EMR and to establish and manage clusters. Amazon EMR-specific commands are part of the extensive feature set of the AWS CLI. It allows you to create scripts that automate cluster management and startup procedures. The best choice if you prefer to operate from a command line is to use the AWS CLI.

SDK (Software Development Kit) offers capabilities for creating and managing clusters that make calls to Amazon EMR. Using them, you can create programs that automate cluster creation and administration. The SDK is best for customizing Amazon EMR. Amazon EMR supports Go, Java, .NET (C# and VB.NET), Node.js, PHP, Python, and Ruby SDKs.

A low-level interface called a Web Service API allows you to utilise JSON to directly call the web service. Creating a custom SDK that calls Amazon EMR is best accomplished by using the API.

Amazon EMR Disadvantages

Complexity:

EMR cluster setup and management can be more complicated than with simpler options like AWS Glue, and it requires some familiarity with the underlying frameworks.