Amazon EMR architecture

HDFS stores data across cluster instances with redundancy to prevent data loss and is ideal for MapReduce workloads

[{"selector":"#anim-c907afb7-2afb-4faf-952e-1cd750152b8f [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c66a14c6-b2f2-447c-a9c2-5e19d2786c3b","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-df80895c-1008-409a-9f4f-57fbc1e27069","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Amazon EMR supports open-source applications with their own cluster management systems, offering flexibility for specific use cases

[{"selector":"#anim-406e553e-a453-42de-88ee-d2aaf00685dc [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-629dabef-c58f-4d5f-8f3f-5304fecd60d3","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-cebf87c5-fc72-4873-bf6e-0c6791548406","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Spark supports libraries like Spark SQL, MLlib, and GraphX, while MapReduce works with Java, Hive, and Pig for data processing

[{"selector":"#anim-862a94d4-79db-4569-aa6b-d2f223f4f433 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-ae539921-cdcd-410b-a998-0d39f34571cd","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-262c7bf4-c02c-4dcc-9af0-9abbd78be5ee","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7a761ad8-303b-4e50-b398-63d093f814c8","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Amazon EMR supports applications like Hive, Pig, and Spark Streaming for tasks like data warehousing, machine learning, and stream processing

[{"selector":"#anim-33099185-9dd1-49f3-8808-3e9003e2460b [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-cc419a95-ad49-4b21-a493-8363eebd7833","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-23f938c2-2da6-42b7-b4e7-555f859bc109","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

A cluster framework that caches datasets in memory and uses directed acyclic graphs for execution, offering faster performance than MapReduce

[{"selector":"#anim-a6b4a4f8-4c6b-4e6e-b664-bc4fb798ad3d [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d641373d-9961-4c70-aeb6-ad3266b5c996","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c5968e81-15cc-4330-957d-937ade52d128","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Amazon EMR uses built-in YARN node labels to assign the CORE label to core nodes, ensuring application masters are scheduled on stable nodes

[{"selector":"#anim-9759771e-c059-4ca1-b6b5-6982748201fc [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-62adc972-2257-4d0a-b3e9-1091280f9797","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-8d91fbbd-e643-4e1a-a413-552a2212d0ee","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

A distributed computing framework that simplifies parallel application development using Map and Reduce functions for key-value pair processing

[{"selector":"#anim-9695c63b-a9ff-4762-a0f8-477c8c4cdafb [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-2cbae2a5-638b-42e2-9d86-36cfe4023cec","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-27617092-c6c9-4b69-87e2-67273139a400","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Amazon EMR ensures job stability by restricting application master processes to core nodes, preventing task failures when Spot Instances are interrupted

[{"selector":"#anim-44d6de16-9dc2-474f-9603-d3a13882737a [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-42e91617-17ff-4e63-b4e6-8178666f20da","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-eba6c54d-dda5-4218-809a-9f7fad2dafbf","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Amazon EMR uses YARN to manage cluster resources and schedule data processing jobs, ensuring efficient resource allocation

[{"selector":"#anim-03fb5556-f55d-4063-b1c2-f27fd50dddf5 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-13a8dc01-c0ec-4526-a745-7ec05f98e69c","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-7f0a1798-a63e-40c0-bc04-b4281dff88c9","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Locally attached instance storage is ephemeral and only lasts for the lifetime of the Amazon EC2 instance

[{"selector":"#anim-c64bb0d9-d95e-4e23-9bc4-8d873d5833ee [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-4ad4cdc1-6078-4792-8ade-814701d849ca","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-ca912eed-f4fa-40a1-bf40-2a9d01adf281","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

For more details visit Govindhtech.com

[{"selector":"#anim-ac6dc246-6b2c-4d53-b60d-60cbbccb594a [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-2e295bf7-5793-4675-8275-b06ea0976380","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-fd667302-3db2-4b0f-9fc6-b27452c2982e","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]