What is the difference between Kinesis vs SQS?
Amazon Kinesis differs from Amazon's Simple Queue Service (SQS) in that it allows for real-time processing of streaming large data.
Kinesis Data Stream
AWS Kinesis Streams offers large-scale data input and real-time streaming data processing. It allows you to order records as well as read and/or replay them in the same order.
Kinesis Data Streams:
- Data can be consumed many times
- Data is deleted after the retention period
- Ordering of records is preserved (at the shard level) – even during replays
- Build multiple applications reading from the same stream independently (Pub/Sub)
- “Streaming MapReduce” querying capability (Spark, Flink...)
- Checkpointing is needed to track the progress of consumption (ex: KCL with DynamoDB)
- Provisioned mode or on-demand mode
- Stores record for 24 hours by default and can retain streaming data for up to 365 days
- Can send stream records directly to services such as Amazon S3, Amazon Redshift, Amazon ElasticSearch, Splunk, AWS Lambda
- Kinesis Provision Shards at 1MB/s produce and 2MB/s consumer
AWS SQS
AWS Simple Queue Service (SQS) offers a robust, highly scalable serverless hosted queue for storing messages and delivering data between application components.
SQS:
- Queue, decouple applications
- One application per queue
- Records are deleted after consumption (ack/fail)
- Messages are processed independently for standard queue
- Ordering for FIFO queues (decreased throughput)
- Capability to “delay” messages
- Dynamic scaling of load (no-ops)
- Can configure message retention period from 1 minute to 14 days, default is 4 days
- Other services can be integrated through AWS Lambda
- SQS does batch at 3000msg/sec and 30,000msg/sec in high throughput mode
Scenarios to use SQS vs Kinesis
SQS can be used in
- Order processing
- Image Processing
- Auto-scaling queues according to messages.
- Buffer and Batch messages for future processing
Kinesis Data Streams can be used in:
- Rapid collecting and processing of log and event data
- Real-Time metrics and reports
- Real Time data analytics
- Gaming data feed
- Complex Stream Processing
- Data Feed from IoT
Top comments (0)