Vertex Gen AI LLM Evaluation Service

This blog article describes the Vertex Gen AI Evaluation Service, demonstrates how to utilise it

[{"selector":"#anim-7dd12a78-1bf2-4832-bbf3-faa6c7b9fda5 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-39.690026891617386%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-c322611f-de02-46ef-bf32-9eb6beebdc21","keyframes":{"transform":["translate3d(114.92064%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-1243a18e-9605-4db8-b4dc-a49f28bf5fa0","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f7c8bb2d-4ac0-44b4-8c01-bba0830df11e","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Making important judgements at different phases of a GenAI application development is necessary

[{"selector":"#anim-dbb1fe68-b331-486e-be1e-ef881e5dd453 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.78483597324875%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-85b51b16-9a5a-4963-a431-057ee0716f33","keyframes":{"transform":["translate3d(114.92064%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b302b1c1-c06b-453c-8fe1-850b69b2e9d8","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-ddde6558-6478-45ce-a704-00fb618b5997","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

The Gen AI Evaluation Service provides developers with both online and offline evaluations using the Vertex AI SDK within their choice IDE or coding environments

[{"selector":"#anim-f6887583-e744-4021-ad71-915f9d62c390 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-33.99621202410176%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-f1ffd107-9414-459e-854f-d3abbefdeb4c","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7acb9ffc-219f-4633-9516-b788942cba84","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e4aaef39-89d2-4592-9615-a6787118e437","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Google cloud provide you with the ability to customise Google cloud extensive library of pre-built metrics

[{"selector":"#anim-35927737-b817-4e64-99c2-a78fa31d84f4 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.249999904429615%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-15927030-3d32-4ac6-809a-6728b86102da","keyframes":{"transform":["rotate(-540deg) scale(0.1)","none"],"opacity":[0,1]},"delay":0,"duration":1000,"fill":"both","iterations":1}]

The code logs evaluation parameters and metrics in Vertex AI Experiments automatically and evaluates a summarization task using metric bundles

[{"selector":"#anim-70fb3fec-eaed-41bb-9155-ed0a1d5a8594 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-37.40671634149489%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-257b6b5b-0897-424e-95ba-d7870b5df093","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d819e764-f1f2-4f9e-b8c7-f57daa4d8e35","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-5f90a3f8-8363-40c2-b871-25aa8180c0e0","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

A prominent Italian insurance company called Generali Italia was among the first to make use of Google cloud Gen AI Evaluation Service

[{"selector":"#anim-58f412fd-3ecb-47ab-b11b-17881d566bdd [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-34.249999904429615%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-68f631a1-dcff-49bd-a6cb-adfc52d65cda","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0a66e035-5002-4dba-8024-5c134d0058db","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-9139a305-4a4e-40b2-9505-04601e240676","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

This innovation speeds up information retrieval by enabling employees to engage in conversational interactions with documents

[{"selector":"#anim-8ca184f5-003b-44aa-97b9-6a33f0638818 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.854838599573288%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-2a6fb085-c0e7-4dea-860f-51f135c08612","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e74773a7-4c9e-4158-a243-c8b1b8b07077","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-152c1522-2a31-4caa-80d2-af3da6888403","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

This capability comes in especially handy when comparing model outputs that are hosted outside of Vertex AI For more details govindhtech.com

[{"selector":"#anim-226c6829-2433-45a3-8ae2-03e99c25a5b8 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-36.92059262580184%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-1306543b-bce4-45a1-a94f-4115274552a2","keyframes":{"transform":["translate3d(115.55555%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-93c03b45-6ff7-493f-8513-faeeb1a5325f","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d9e23f7b-c03e-49e0-b03f-8712cc22b15c","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]