GKE: Your batch platform’s new residence


This is the first of a series of posts explaining why administrators and architects should to use Google Kubernetes Engine (GKE) to create batch processing systems

Innovation is happening more quickly now that Kubernetes is a top container orchestration technology for managing and delivering containerized applications

[{"selector":"#anim-c923ef36-e4ec-4669-beea-4c7c13a80136 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.499999772451451%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-8f6df3f0-88ed-428a-954f-3bfc394a1d7f","keyframes":{"transform":["translate3d(120.61855%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-2260b301-e86c-40e3-9b5e-061751435ba7","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

GKE is a managed Kubernetes solution that expedites users’ time-to-value while affordably abstracting the underlying infrastructure

[{"selector":"#anim-6f181815-a2a7-4a6b-81cd-daedbce8184c [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.241477158901287%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-41d33457-d65d-4007-858e-71731c5a0b65","keyframes":{"transform":["translate3d(-141.15045%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-5fb138da-8c40-4fda-9d4a-fc8051b4ed5b","keyframes":{"transform":["rotateZ(-180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

The Kubernetes Job API allows you to manage Jobs A Job starts one or more Pods and keeps trying to run them until a predetermined number of them finish successfully

[{"selector":"#anim-3e4439d1-3ac1-4f5f-bb99-b344c1f76760 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.249999904429615%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-42a7d087-0328-45c4-b4a2-d32282b2a792","keyframes":{"transform":["scale(1)","scale(1.5)","scale(0.95)","scale(1)"],"offset":[0,0.33,0.66,1]},"delay":0,"duration":1450,"easing":"ease-in-out","fill":"both","iterations":1}]

The Job records the successful completions of Pods as they happen The job or assignment is deemed finished after a predetermined number of successful completions

[{"selector":"#anim-562e1fad-b36a-47b4-860b-2f19a5610024 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.73900059627012%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-7cdc5dee-6b8d-4251-82ed-95c62c66cd43","keyframes":{"transform":["translate3d(126.96628%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-28c1d2c0-205a-4df5-b81e-7fb82c369678","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-2b274087-a98e-4884-bd71-ece22baa8eef","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Nodes, or Compute Engine virtual machines (VMs), are joined together to form clusters in a GKE environment

[{"selector":"#anim-ed6ad987-84a6-46d4-b8c4-a6f58ec23cab [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(7.729800306022057%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-b0939012-8856-4bad-9748-4270ccee10eb","keyframes":{"transform":["translate3d(-131.85841%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d20bba29-a2d5-45e1-b35e-532d18a64411","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-946ec7e1-d3ef-4419-b06c-9bb9cc6ee4da","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

The biggest Kubernetes clusters that a managed provider can presently host are supported by GKE. For many bulk use cases, a cluster size of 15,000 nodes

[{"selector":"#anim-dee88b64-2e6b-4b2b-b2d2-d64f469ed169 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(14.843749786673241%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-942fd69d-d681-44b6-8827-9ba20af60a9d","keyframes":{"transform":["translate3d(146.08697%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-ad4e83fb-8c9c-455a-8a5a-ae333fb158ff","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

An option to managing several single-tenant clusters and access control is GKE cluster multi-tenancy According to this approach, several users and/or workloads referred to as “tenants” share a multi-tenant cluster

[{"selector":"#anim-43109ab8-a94e-4674-a3ed-27cb2b2f2594 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(34.179687404002955%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c2ffc789-484e-44fb-a6e4-af781fdcd8b5","keyframes":{"transform":["scale(1)","scale(1.5)","scale(0.95)","scale(1)"],"offset":[0,0.33,0.66,1]},"delay":0,"duration":1450,"easing":"ease-in-out","fill":"both","iterations":1}]

You have control over which logs and metrics, if any, are delivered from your GKE cluster to Cloud Logging and Cloud Monitoring thanks to GKE’s integration with Google Cloud’s operations suite

[{"selector":"#anim-92f43e45-5532-4cd7-812f-6df5c7dfd17c [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-b27e8a50-1579-4b50-a73b-0f428379f4de","keyframes":{"transform":["translate3d(-128.13689%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7cb7e7ad-78d2-4b25-8872-7ba156e97db0","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-2dea85a0-2a4c-44ed-8269-a0d551f6e8ff","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]