Amazon EMR

Amazon EMR simplifies running big data frameworks like Apache Hadoop and Spark on AWS for business intelligence and analytics

[{"selector":"#anim-a33c47a5-b770-4173-a13d-a5e2310d0e26 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-3ba2d4ea-36e0-4e2c-8469-5bb193ef1835","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-7b061d16-b85f-47d6-bc3b-66ce58c2ebfc","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Jobs can be submitted as steps during cluster creation, through the EMR UI, API, or CLI, or interactively via SSH to the primary node

[{"selector":"#anim-91f6a4a0-caec-4a94-a20e-3ce354d392dd [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-b46d734a-184c-44dc-91a7-60854c7c982b","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-82a511ab-6eda-4eb5-aa04-cb1756b24834","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Data is processed in sequential steps, with each step performing specific tasks like data manipulation or querying

[{"selector":"#anim-fce338bf-b3fd-4e94-984f-c2a1f16c254f [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-f4823650-0032-42e1-aac6-cda072985431","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-5c5d7462-a07d-4946-a552-130eb11b3ee9","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Data is typically stored in HDFS or Amazon S3, processed through steps, and output to a designated location like an S3 bucket

[{"selector":"#anim-8634abef-229d-4a02-91ab-cd5574f8bb7f [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-bd661877-a228-4b3d-a85e-d037fe2ca9ae","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-328558bc-ba58-4efc-8bde-94575c28f92f","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Steps transition through states (PENDING, RUNNING, COMPLETED, or FAILED), with options to cancel, retry, or terminate the cluster on failure

[{"selector":"#anim-6cc9421c-573e-49d8-a96c-df159c9f637b [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-f0538a6a-d9cf-4321-affc-d97bbf1ef3ee","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-2454bd29-0823-46ab-9772-3839b4eafc2c","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Clusters progress through states: STARTING, BOOTSTRAPPING, RUNNING, WAITING, TERMINATING, and TERMINATED

[{"selector":"#anim-7105cb23-a13d-4f0b-9c7c-4dbfbad51700 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-00b61e73-2d9c-4d0b-952a-a34222ca1366","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b798cd07-6696-4b14-a0b7-f941bda91ecc","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-a69204d2-50ab-49bb-8902-914160d52b34","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Custom scripts can be executed during cluster setup to install additional software or configure instances

[{"selector":"#anim-5925575b-98ba-4005-a130-3f4a33f24a69 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-e54a4cf4-0556-4dec-84c8-a5f736ece6e7","keyframes":{"transform":["translate3d(-115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-58cacb83-5eac-4cfb-8634-0c0aabf0882b","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-a7d01bdf-dea5-45ed-913c-aa33f17823f0","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

EMR supports pre-installed applications like Hive, Hadoop, and Spark for data processing

[{"selector":"#anim-756007c0-3dd7-4c44-80dd-e7078b71015c [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-7137b4a8-2a27-4dcc-bb05-e2d8cd514a0b","keyframes":{"transform":["translate3d(-115.2381%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7d6bc455-845e-4303-8d3e-efb78ccdc002","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0b5d989f-b150-4514-aecb-97168c8ca124","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Clusters can auto-terminate after completing steps or remain in a WAITING state for manual shutdown

[{"selector":"#anim-1f96bc82-1822-48f7-aa6b-b4514b703047 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-80e85ccb-7603-4d55-aae1-2a7ca402bcf2","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d9a871b1-af1e-4635-9507-f4b4bc7a8a00","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

EMR supports custom AMIs, hardware configurations, and termination protection for enhanced control and recovery

[{"selector":"#anim-f64e45cd-76f7-4d99-a86e-e423e600e2d0 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c5524d9a-4cc5-4180-a2b6-3690412953fc","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-07449964-d429-4605-ad88-ce3b2adf4a43","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

For more details visit Govindhtech.com

[{"selector":"#anim-e60e1719-eb37-4ff4-b740-2ae9372b8011 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]