Real-time Text Analysis: BigQuery‘s Cutting-Edge Tools

BigQuery‘s sophisticated text analyzers and preprocessing tools contain enormous volumes of structured and unstructured text data, including information about customers and company activities

[{"selector":"#anim-f4eb9432-af95-45df-ab9b-e41a62635e38 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-28.93339755556636%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-9fd7a933-0ebb-48d0-972f-7c53ef07d51d","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0086142e-bc06-4a4b-85f8-5559c09ba4e1","keyframes":{"transform":["translate3d(-114.50152%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Text preprocessing, which converts unstructured or raw natural language into formats that are machine-readable, is an essential stage in pipelines for Text Analyzer and information retrieval

[{"selector":"#anim-46a91c65-8d3a-4dae-b7de-51e078fe7480","keyframes":{"transform":["translate3d(-115.14196%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-ed5f0e29-f7b0-4536-ad88-c3b42286ba8f","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b2928261-cff9-4e62-9867-2186e8edf0d1","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-c429db26-ce79-4335-976d-5c3e479f13ee","keyframes":[{"offset":0,"transform":"translate3d(0, -429.12466%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.29,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.45,"transform":"translate3d(0, -120.669854392%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.61,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.71,"transform":"translate3d(0, -41.024317496%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.8,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.85,"transform":"translate3d(0, -15.405575294%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.92,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.96,"transform":"translate3d(0, -6.694344696%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":1,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"}],"delay":0,"duration":600,"fill":"both"}]

In many instances, the quality of the tokenization method used has a significant impact on how successful a text search index

[{"selector":"#anim-a855adfe-8025-4a14-afc5-1f0cd313386f","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0fb6c0dd-c594-4faa-ad02-5e8c357ff496","keyframes":{"transform":["translate3d(0px, -153.63984%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

The procedure entails obtaining entries from a logs table, which is produced by routine company operations, that include pertinent customer information

[{"selector":"#anim-2610d2ea-5686-45be-a223-95771b4f30bf [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(28.93339755556636%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-ee9cbb2b-9cd2-4866-b12e-8e1faf02ab0b","keyframes":{"transform":["translate3d(123.48755%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-d0c25fe7-d8ef-4332-abfe-0ced26e72aa4","keyframes":{"transform":["rotateZ(180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

Before building our search index which may be a costly and time-consuming procedure They experiment with several configurations using the recently introduced TEXT_ANALYZE function

[{"selector":"#anim-11e33e0b-b15f-4101-bdfa-6005a95f3fdc [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-21.874999829338588%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-b67dcac8-dbad-4e91-bde6-b5f14af6116a","keyframes":{"transform":["translate3d(-121.60278%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e6a31185-a2c2-4bd1-aa70-b7541469f320","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-36e20520-4d4d-48d5-a567-7f9aa9342310","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

To refresh your memory, the SEARCH function operates by applying the analyzer (with the given configuration) to the search data and the input search query

[{"selector":"#anim-5ede2161-27d7-403f-bd63-28fc04740b94 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(34.249999904429615%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-321dc649-43d6-4edc-a299-9aa4c57f6523","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-adaafcd4-8bb5-411b-a453-b7afc0b70863","keyframes":{"transform":["translate3d(0px, -125.88911%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

They can see from this example that the new PATTERN_ANALYZER is a useful tool for assisting in the creation of an efficient search index for our hypothetical fraud investigation

[{"selector":"#anim-017ae838-db26-4105-9923-2531dfad0727 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.35190206075711%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-ac9c729d-b813-4460-891d-e554d394e554","keyframes":{"transform":["translate3d(-125.83026%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-f477c0b9-6006-42fe-aa3a-97ca347a0fa6","keyframes":{"transform":["rotateZ(-180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}]

Our text analysis toolkit has been enhanced with new capabilities that significantly improve its current functionality For more details visit govindhtech.com

[{"selector":"#anim-5cb24548-5e21-4f4c-bab5-6122d0af2458 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(28.93339755556636%, 0, 0)","translate3d(0%, 0, 0)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-9399ae1c-4746-4653-873d-9369874c89f5","keyframes":[{"offset":0,"transform":"translate3d(0, -147.0178%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.29,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.45,"transform":"translate3d(0, -41.341405359999996%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.61,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.71,"transform":"translate3d(0, -14.05490168%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.8,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.85,"transform":"translate3d(0, -5.27793902%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":0.92,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"},{"offset":0.96,"transform":"translate3d(0, -2.2934776799999996%, 0)","easing":"cubic-bezier(.5, 0, 1, 1)"},{"offset":1,"transform":"translate3d(0, 0%, 0)","easing":"cubic-bezier(0, 0, .5, 1)"}],"delay":0,"duration":600,"fill":"both"}]