AWS Glue Studio: Code-Free Data Pipelines

The fully managed ETL solution AWS Glue makes data preparation and loading for analytics easy

[{"selector":"#anim-0a976e8a-d68c-441b-8270-420c9a2c3aa8 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-38.71822027052565%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-235e476c-faca-40aa-9518-30d13f1ca937","keyframes":{"transform":["translate3d(-115.18988%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-82e3a365-7df6-4990-ab49-d5712be8335e","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-61f9934a-1ea9-435c-a510-8c47598f73be","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

AWS is pleased to inform that AWS Glue Studio Visual ETL now offers data preparation authoring on a broad basis

[{"selector":"#anim-8c2a074d-4aa8-49c8-aa7f-815300e0b1eb [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-37.44419635238331%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-2d900fcc-2437-497e-983a-c263251e56f7","keyframes":{"transform":["translate3d(-121.73913%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e0fd5ccc-0b7e-4c98-98cd-1424393aa739","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-28f6482b-a7f9-4047-ad42-93403fa2f5e9","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Scripts created for the Ray and Apache Spark runtime environments can be executed by jobs

[{"selector":"#anim-202d04cd-d906-4e3a-9f09-d1390c917431 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-35.89313871566194%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-840c3ed3-b45d-4776-8e23-ac7bd404e6f2","keyframes":{"transform":["translate3d(-114.87342%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-bb6183c7-cdeb-4e67-b56c-26abd732b76e","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f5bd55f4-ae08-46d6-b408-e2732bd7b30a","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

The AWS Glue Studio code generator can generate an Apache Spark API (PySpark) script automatically given a source schema and a target location or schema

[{"selector":"#anim-7cc53042-2aa4-46f1-bb63-eee593f234d2 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-33.12499989760315%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-124fa878-a854-461c-9d96-b9301acd5673","keyframes":{"transform":["translate3d(-116.16162%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-ddca280f-947a-4d05-aa46-a6a3ff719420","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7420efd8-66b5-40fe-a4db-45cfda4928b5","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

After logging into the AWS Management Console to examine the jobs that are currently in progress

[{"selector":"#anim-40d3a5a2-98b2-4729-9fb1-eede14f4684d [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-36.79496943206751%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-62ff6437-e0df-4649-a45a-f1b6d6c7eec5","keyframes":{"transform":["translate3d(115.18987%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d1f57e73-c26e-47b8-bc53-e3cf4adefe5c","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-624805e1-ae56-42c4-a01f-8b16ba3b6d54","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

This can be accomplished by modifying the job script in developer mode or by modifying the nodes in the visual editor

[{"selector":"#anim-99298294-fd23-42d1-b06d-1b2fcdb061fa [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-35.163834861431035%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-b2d39068-2482-4e14-b109-e6a3ad9eba62","keyframes":{"transform":["translate3d(120.52980%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-91c4a303-390b-4294-b068-4d2e0bb0a419","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-6f1f75dd-d4b8-4c98-83fc-a747d95b878a","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Data engineers can specify connections to the data and configure the data flow process’s ordering using the visual flow-based interface in Glue Studio

[{"selector":"#anim-4f882aba-620c-4891-b563-947c10c8c554 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-2e7551d7-39b8-43c7-8745-8202483a5c7b","keyframes":{"transform":["translate3d(115.18987%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-72cdf361-1239-4d04-83df-0e6279f233f9","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-8979aa3c-ff2d-4a56-ac77-014081a51b50","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Select Author Recipe to begin an authoring session and add transformations when the data frame is finished, once the data preview session is ready For more details govindhtech.com

[{"selector":"#anim-4db7e169-dd21-4bc6-a356-0062f1b4b8e5 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-41.925837271580455%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-0c90986c-8e12-4f40-9111-22c3bec41164","keyframes":{"transform":["translate3d(115.50632%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-3e884593-8e77-4777-9a04-7f76cefa44c5","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-829426da-857c-438b-ad0c-53b400e15dd3","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]