Well I'm surprised to hear "could not scale by design". We are in the works of implementing it for our enterprise and have seen the exact opposite. Straight from the Apache website: "The disk structures Kafka uses scale well—Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server." The logs are written to disk in conjunction of using RocksDB for certain cached data storage.
Well I'm surprised to hear "could not scale by design". We are in the works of implementing it for our enterprise and have seen the exact opposite. Straight from the Apache website: "The disk structures Kafka uses scale well—Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server." The logs are written to disk in conjunction of using RocksDB for certain cached data storage.
Yes, I was surprised as well.