Gearpump is a real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks. Gearpump is event/message based and featured as low latency handling, high performance, exactly once semantics, dynamic topology update, Apache Storm compatibility, etc.
The name Gearpump is a reference to the engineering term "gear pump," which is a super simple pump that consists of only two gears, but is very powerful at streaming water.
Gearpump Technical Highlights
Gearpump's feature set includes:
- Extremely high performance
- Low latency
- Configurable message delivery guarantee (at least once, exactly once).
- Highly extensible
- Dynamic DAG
- Storm compatibility
- Samoa compatibility
- Both high level and low level API
Per initial benchmarks we are able to process 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.