The latest DirectML accelerates AMD GPU AWQ-based LM

Minimize Memory Usage and Enhance Performance while Running LLMs on AMD Ryzen AI and Radeon Platforms Overview of 4-bit quantization

[{"selector":"#anim-ffbf948a-7648-4ad9-bbaf-d2c95e9ce2c7","keyframes":{"transform":["translate3d(111.67192%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-aa606b09-9f0b-49f7-82cd-535c29e291d2","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-4eae4441-7178-4926-8b10-d653271b7fa9 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-33.882521392171114%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

Over the past year, AMD and Microsoft have collaborated to accelerate generative AI workloads on AMD systems utilising ONNXRuntime with DirectML

[{"selector":"#anim-f383d707-c593-41ad-9bf5-90e9496fc59e [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-ba959f46-cee8-44a6-9e6e-e4616ce6b30e","keyframes":{"transform":["translate3d(-138.18897%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-d153a2b1-5366-497b-a33b-902d828ded45","keyframes":{"transform":["rotateZ(-180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

The number of LLM parameters (7B, 13B, 70B, etc.) greatly increases system memory consumption, making workload management difficult

[{"selector":"#anim-4bf2d0b5-2522-407d-b036-441ffaef5a89","keyframes":{"transform":["translate3d(116.78321%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-3d79dd9a-eb97-409f-bd3a-544e5a46de12","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-aef44731-dee9-48ff-815b-d4382c8a353b [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-33.52855041249698%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

Microsoft and AMD are thrilled to offer AWQ-based LM acceleration on AMD GPU architectures in the newest DirectML and AMD driver preview

[{"selector":"#anim-2acbdb8b-1cef-4060-b153-38ebc1308ef7","keyframes":{"transform":["translate3d(123.37166%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-b2bcb431-44fe-4b48-9505-038a2323e211","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-6249beb8-9e69-4e0e-9fb0-bbe01e60e8f0 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.249999886225726%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

When possible, AWQ reduces weights to 4-bit without impacting accuracy. This significantly decreases LLM model memory and boosts speed

[{"selector":"#anim-f940a307-73ef-45be-bd16-34e09262fba3","keyframes":{"transform":["translate3d(129.58333%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-63717d60-f72c-416a-9127-fb0310a2d366","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-8de9a9c6-30a7-49d6-9f88-ab6f993e537f [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-35.234374910402764%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

AMD driver resident ML layers dequantize parameters and accelerate on ML hardware during runtime to increase AMD Radeon GPU performance

[{"selector":"#anim-aced6c68-93e4-417f-aeda-d8dafa028869 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-dc7deafb-4057-4746-b8fb-059e3b501948","keyframes":{"transform":["rotate(-540deg) scale(0.1)","none"],"opacity":[0,1]},"delay":0,"duration":1000,"fill":"both","iterations":1}]

This 4-bit AWQ quantization is carried out utilizing Microsoft Olive toolchains for DirectML

[{"selector":"#anim-7271c1a0-d5cb-43cc-9eb2-69106d7ff3ca [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(3.552713678800501e-15%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-3cedfaee-44c8-4d45-b84b-18747ad52b44","keyframes":{"transform":["translate3d(152.99999%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-3989f436-2c94-4c47-899a-3387363690f6","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

This method makes it possible to execute language models (LM) on a device with limited memory

[{"selector":"#anim-f99a41f1-7225-46c3-8fb2-58f07f417b13 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-d3ce1bfb-0ec3-4550-a3e3-6a25d1301930","keyframes":{"transform":["translate3d(159.99999%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-8247c2e0-096e-4c27-85cd-82df2524e1a4","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

AMD Ryzen AI platforms with AMD Radeon 780m have a memory footprint equal to 16-bit weight00000s AMD Radeon 7900 XTX computers For more details Govindhtech.com

[{"selector":"#anim-d6ec2116-c056-45e2-8f83-e5496cf07220 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-f46fcc43-ffe9-4e96-aeb8-ef8fd88d949c","keyframes":{"transform":["translate3d(-124.54873%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-969e9e37-1533-4d50-858a-b34df3493bf4","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-6a1403c2-ca77-49a3-892a-1c69c521132b","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]