Google Colossus Storage: Getting SSD Speeds At HDD Costs

Google's all-purpose storage platform, Colossus, has object storage management and scalability, an intuitive programming style used by all Google teams, and throughput that rivals the best parallel file systems

[{"selector":"#anim-e3d2a893-b784-4a89-b177-2072c8e1c0ef","keyframes":[{"offset":0,"transform":"translate3d(0, -241.39853%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.29,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.45,"transform":"translate3d(0, -67.881266636%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.61,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.71,"transform":"translate3d(0, -23.077699468%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.8,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.85,"transform":"translate3d(0, -8.666207227%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.92,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.96,"transform":"translate3d(0, -3.7658170679999996%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":1,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"}],"delay":0,"duration":600,"fill":"both"}]

Hyperdisk ML serves 2,500 nodes @ 1.2 TB/s with a Colossus SSD. Colossus, Spanner's tiered storage feature, combines cheap HDD storage with fast SSD storage in the same filesystem

[{"selector":"#anim-b8a6953f-c381-40b2-8b82-9843c4320bc4 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Colossus SSD caching allows Cloud Storage to provide the cheapest storage while supporting AI/ML applications' demanding I/O. Finally, BigQuery's Colossus-based storage speeds up huge searches

[{"selector":"#anim-8db4828b-35c9-4937-ae81-becc416d90e2","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-222ab558-783f-4f29-ab58-dffe6a7dd156","keyframes":{"transform":["translate3d(0px, 135.99313%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Colossus reduced the complexity of the GFS programming model to an append-only storage system that combines the scalability of object storage with the well-known programming interface of file systems

[{"selector":"#anim-0855fce6-ebc9-48ec-9b5a-977ba216d944","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-ef29af01-bc98-4fad-9233-8d980f314feb","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

The Colossus metadata service is composed of “custodians,” who maintain disk-space balance and data durability and availability, and “curators,” who handle interactive control tasks like file creation and deletion

[{"selector":"#anim-8840c59a-1c6e-403b-be1f-63647481160d [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-12.514648210040342%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

Colossus storage clients store data directly on “D servers,” which house its HDDs or SSDs, after interacting with curators for metadata

[{"selector":"#anim-b1ddcbde-059c-4601-b6e1-e123729a1393","keyframes":{"transform":["translate3d(128.45528%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-a7c654ef-d3d4-4c6b-b5a6-26d04b86ce3c","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-8ea17490-a418-488c-9900-8cdc5fe64bbc","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

A Google Cloud zone’s core building block is a single Colossus filesystem, which Google constructs for each cluster

[{"selector":"#anim-c9a6558c-bc03-44f5-afcb-60c8b50d0887","keyframes":{"transform":["scale(1)","scale(1.5)","scale(0.95)","scale(1)"],"offset":[0,0.33,0.66,1]},"delay":0,"duration":1450,"easing":"ease-in-out","fill":"both","iterations":1}]

Many of Google's largest filesystems exceed 50 TB/s read and 25 TB/s write throughputs. This bandwidth could send 100 8K videos per second

[{"selector":"#anim-c8d248cb-7d2e-44b3-99a9-b134bfdde6a8 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(12.514648210040342%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}]

It's hard to read 50 TB/s on sluggish discs with all your data. Two major new features in Colossus are SSD data placement and caching, driven by “L4”

[{"selector":"#anim-1c7c864b-d12e-4aab-ac40-ea3d62a46349","keyframes":{"transform":["translate3d(137.87236%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f8df4b3a-2beb-4dcc-8169-4750640addaa","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-c21c2a55-4ad3-4a48-9cd4-8791a7e65d88","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Advanced users can utilise “hybrid placement” to tell Colossus to keep only one replica on the SSD: ssd.1/myfile /cns/ex/home/leg/partition. This solution is cheaper, but HDD lag occurs if the D server hosting the SSD copy is down

[{"selector":"#anim-cbab9c89-a12a-4597-ba0a-d8d77556ca43","keyframes":{"transform":["scale(1)","scale(1.5)","scale(0.95)","scale(1)"],"offset":[0,0.33,0.66,1]},"delay":0,"duration":1450,"easing":"ease-in-out","fill":"both","iterations":1}]

In reaction to cache misses, L4 may add accessed data to SSD cache. It does this by ordering an SSD storage server to shift HDD data. L4 eliminates items when the cache fills to make place for new entries

[{"selector":"#anim-4bb59d69-ccc6-4129-875e-53478db103fd","keyframes":[{"offset":0,"transform":"translate3d(0, -227.60356%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.29,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.45,"transform":"translate3d(0, -64.002121072%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.61,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.71,"transform":"translate3d(0, -21.758900336%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.8,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.85,"transform":"translate3d(0, -8.170967804%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.92,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.96,"transform":"translate3d(0, -3.5506155359999996%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":1,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"}],"delay":0,"duration":600,"fill":"both"}]

Google Cloud use these I/O patterns to simulate placement requirements like “place on SSD for one hour,” “place on SSD for two hours,” and “don’t place on SSD” online. This scenario helps L4 choose the best policy for each group

[{"selector":"#anim-b7d9e1b2-b518-443c-b8c8-5373fa567694 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Google Cloud relies on Colossus storage to serve billions of users. Advanced SSD positioning features dynamically respond to workload variations to lower costs and boost performance

[{"selector":"#anim-84c5793d-7a6c-4db8-9c9d-692fc906150e","keyframes":{"transform":["translate3d(124.72728%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-6cd8e2a7-5a4f-4288-acaa-ce04befdba8f","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]