Gluten And Intel CPUs Boost Apache Spark SQL Performance
By shifting JVM data processing to native libraries, Gluten may benefit from the speed improvement that Intel processors can provide with native libraries that are optimized to instruction sets
Gluten uses a vectorized SQL processing engine called Velox (Meta’s open-source) C++ generic database acceleration toolkit to improve data processing systems and query engines
The execution plan is constructed, loaded, and handled effectively by the native engine (which also manages native memory allocation) before being sent back to Gluten as a Columnar Batch
The execution plan is constructed, loaded, and handled effectively by the native engine (which also manages native memory allocation) before being sent back to Gluten as a Columnar Batch
Gluten employs a shim layer to support different Spark versions and a fallback technique to execute vanilla Spark to handle unsupported operators
While outsourcing as many compute-intensive data processing components to native code as feasible, the Gluten plugin makes use of Spark’s own architecture, control flow, and JVM code
Enhancements in Performance Was ObservedThis section examines test findings that show how performance may be enhanced by using Gluten in your Spark applications
Enhancements in Performance Was ObservedThis section examines test findings that show how performance may be enhanced by using Gluten in your Spark applications
Incorporating Gluten into your environment will help you get more out of your technology and reduce time to insight if your data center has servers of this generation