BFloat16 Turbocharges AMD GPUs Unleash the Future!

When creating machine learning models, the vast majority of machine learning (ML) engineers make use of the single precision (FP32) datatype

[{"selector":"#anim-0cb1d7ba-6a4b-4f3d-b785-fc8d8e02a3d9","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-5d5a13cf-8f9f-4a32-b96a-4e5091b6bb02","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-78b968e2-2bbc-48f4-9975-6325a4076181 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.08937242519085%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

TensorFloat32 (TF32), a drop-in replacement for FP32-based models, has lately gained popularity and is becoming more widely used

[{"selector":"#anim-d7c70d22-33de-420b-b547-56d426c8b4a9","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-fd465b91-5996-4510-92c5-86ac3a1b76bf","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-8b23a4b6-aac2-4ded-a2d0-4751f8bc42ab [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.179687404002955%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

An application that already makes use of the TF32 infrastructure would be able to notice acceleration while using this strategy, and it would do so without requiring any extra code modifications

[{"selector":"#anim-2b82c47b-611e-473e-8c75-ea9c74c32678 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-28.90624987200394%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-53a876cd-3e38-4b4f-abf4-4f21d4e7e8a4","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-6c0208aa-3ea7-4647-9e96-947df939bfcb","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Specifications of the Methodology for Implementation

[{"selector":"#anim-3ee3e637-7e9e-499b-9f87-e5d3d8a12d2a [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-28.90624987200394%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-143f88c1-3819-4a93-acc8-ec8ae95bec1b","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-1e3711e7-ec4f-478b-8ccf-57ea11fe279e","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Pytorch now supports three different levels of precision for FP32 models

[{"selector":"#anim-839dd7b8-3b0a-4eb5-99bc-13ac0fd5c9d0 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.179687404002955%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-4c400613-78dc-4ec3-a187-8d8d270c44c9","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c2000e74-163c-406d-8644-8b8cd5803c07","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Pytorch’s Linear layers employ TF32-emulation in its present implementation; this was chosen because of its superior performance

[{"selector":"#anim-d64da813-a336-4906-b4fd-5abfe4a5c4a7 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.284090795523493%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-a836b999-1ec6-4ea5-a228-a87df9404885","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-549ff78d-0bf2-40f7-b4c6-10dbd77df6c0","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Examining the Differences in Performance

[{"selector":"#anim-8ef902df-12ce-4465-854c-d1d8f2d890be [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(34.179687404002955%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-68eb2bcd-2db4-4d58-923c-91d087854890","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-137e5c70-59de-4b8c-a8a1-216d5dab1218","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

we see a speedup of improvement to 1.79 times over the default implementation on Transformer generated from MLPerf implementation For more details Govindhtech.com

[{"selector":"#anim-940988ba-48d5-4488-886b-53b7f493cc69 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(35.2812499106872%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}]