A Formula 1 Team used Pathway as a core real-time processing engine to:
- Power real-time streaming use cases, through a fast, flexible, and user-friendly data processing system.
- Enable end-users to create their User-Defined Functions (UDFs) independently, to feed the various business needs.
Why They Chose Pathway
- Ability to support batch and streaming data processing tasks
- Ability to perform advanced data transformations, including ML with small latency.
- Performance: low latencies for high data throughputs, up to 120 Hz.
- Capable of processing millions of data points per second, it largely surpasses current reference technologies such as Spark (in both batch and streaming), Kafka Streams, and Flink.
- Benchmarking of WordCount and PageRank against the above also found that Pathway supports more advanced operations and is up to x90 faster thanks to its maximised throughput and lower latency. The benchmarks were stress tested by the developer community, are publicly available, and can be run by anyone. (https://github.com/pathwaycom/pathway-benchmarks)
- Full platform approach, with a possibility of a front-end integration downstream
- A solution that can adapt to the many workflows that co-exist across business units
- Possibility for the code to be maintained and versioned, and get an end-to-end code visibility of the data pipeline
- Ease of programming, to enable engineers to easily implement User-Defined Functions (UDFs)
- Pathway is made to be accessible to users of the Python/Pandas/scikit data stack.
- Did you know? Data processing code is written in Python, and Pathway compiles it into a Rust dataflow. All I/O operations in Pathway are performed using highly-optimized Rust libraries and native Rust data parsing and formatting. There are multiple stages of optimization on this dataflow to ensure that the basic data transformations are being used as efficiently as possible. Whenever the user writes a user-defined function in Python, it will be converted into Rust to allow for easy parallel and distributed execution, and speed. For functions that remain native in Python, e.g. API calls or use of external libraries, the Pathway engine ensures that the Rust-to-Python bridge is streamlined and fast.
- A solution that can be industrialized and put into production easily