WILMINGTON, DE — The Apache Software Foundation (ASF) has announced that Apache® DataFusion™ is now a Top-Level Project (TLP). This milestone marks a significant achievement for the fast, extensible query engine designed for building high-quality data-centric systems in Rust. DataFusion uses the Apache Arrow in-memory format, which enhances its performance and scalability.
What Is DataFusion?
DataFusion aims to be the go-to query engine for new, fast data-centric systems, including databases, dataframe libraries, machine learning, and streaming applications. By leveraging the features of Apache Arrow and Rust, DataFusion allows projects to focus on developing unique features rather than reimplementing standard ones. These include expression representation, standard optimizations, parallelized streaming execution plans, and file format support.
The engine can be used without modification as an embedded SQL engine or customized as a foundation for building new systems. It is already employed in various high-performance systems, from specialized analytical databases like Apache HoraeDB to new query language engines and research platforms.
Key Features
Apache DataFusion offers several standout features:
- Fast Execution: The engine is vectorized, multi-threaded, and supports streaming execution, allowing for high throughput and low latency.
- File Format Support: It supports Parquet, CSV, JSON, and Avro through built-in plugins and custom file formats via extension traits.
- Extensibility: Users can define custom scalar, aggregate, and window functions, data sources, SQL queries, custom plan and execution nodes, optimizer passes, and more.
- Advanced Query Optimization: The engine includes state-of-the-art optimizations such as expression coercion, projection and filter pushdown, sort and distribution aware optimizations, and automatic join reordering.
- Asynchronous I/O: It supports streaming input/output from popular object stores like AWS S3, Azure Blob Storage, and Google Cloud Storage.
- Cross-Language Plans: DataFusion supports Substrait, enabling it to pass plans across different languages and systems.
- Implemented in Rust: The use of Rust ensures safety and performance, making the engine reliable for critical data processing tasks.
Industry Impact
Andy Grove, Apache DataFusion PMC Member and original creator of DataFusion, highlighted the project’s rapid growth. “What started as a modest project to provide a simple and efficient query engine has evolved into a robust, high-performance system that powers data-centric applications worldwide. Becoming a Top-Level Project is a significant milestone, and I am excited to see how the project will continue to innovate and shape the future of data processing,” Grove said.
Broad Adoption and Use Cases
DataFusion is already fueling a range of applications:
- Analytical Databases: Systems like Apache HoraeDB use DataFusion for high-performance analytics.
- Query Language Engines: Projects such as prql-query have adopted DataFusion for its versatility.
- Research Platforms: New database systems like opt-d rely on DataFusion for experimental development.
- Streaming Data Platforms: Companies like Synnada leverage the engine for real-time data processing.
- SQL Support: Libraries like dask-sql use DataFusion to enable SQL queries on large datasets.
- File Processing Tools: Tools like qv utilize DataFusion for reading, sorting, and transcoding files.
- Spark Replacements: Solutions like Comet and Blaze employ DataFusion as an alternative runtime.
Paul Dix, CTO and co-founder of InfluxData, emphasized DataFusion’s role in their recent product. “DataFusion’s capabilities have been integral to the development of InfluxDB 3.0. By building with and contributing to this project, we’ve been able to deliver a powerful, vectorized SQL engine to our users, all while benefiting from continuous improvements from a dedicated global community,” Dix said.
DataFusion: Pioneering High Performance & Scalability in Open-Source Data Solutions
DataFusion’s promotion to a Top-Level Project reflects its maturity and the trust the developer community places in it. The engine’s ability to deliver high performance and scalability makes it a crucial tool for enterprises dealing with large volumes of data. Its extensibility and flexibility allow for a wide range of applications, from real-time analytics to complex machine learning models.
This development also highlights the growing importance of open-source software in driving technological innovation. By fostering collaboration and continuous improvement, projects like DataFusion accelerate advancements in data processing, benefiting businesses and consumers alike.
As DataFusion continues to evolve, it promises to set new standards in the field of data-centric systems, paving the way for even more sophisticated and efficient data solutions.
For the latest news on everything happening in Chester County and the surrounding area, be sure to follow MyChesCo on Google News and MSN.