Just like Facebook's Scribe tool, Kafka can be used for processing large amounts of streaming data.
It can basically handle all kind of activity stream data and processing on a consumer-scale website.
This activity includes page views, searches, and other user actions, all important ingredients for a social website.
Here are some key features of "Apache Kafka":
· Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
· High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
· Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
· Support for parallel data load into Hadoop.
What's New in This Release: [ read full changelog ]
· Kafka now supports block level compression. See compression wiki for details. Currently only gzip is sup ported.
· Log retention can now depend on space (log.retention.size).
Find us on Google+
