[SAA + SAP] 22. Kinesis & AWS MQ
SAA
- Shared 2 MB/sec, all shards shared 2MB/s
- Enhanced: Per shard 2MB/sec
- Max retiention 365 days
- Can replay data
- same partition goes to same shard
- Serverless
- Near real time
- Each record 1MB max
- S3 backup
- Destionations: S3, Redshift, ElasticSearch
- Stream can be replayed
- Stream is realtime
- Firehose can do transformation using Lambda
- For SQS, use GroupId in order to group message together
- Same patition key goes to same shard
Kinesis: max number of consumer = shard number
SISO: 100 GroupID = 100 consumers
- STOMP, AMQP, MQTT protocols
- Migrating application to cloud
- Has Failover
- Using EFS as storage
SAP
-
Kinesis data is automatically replicated synchronously to 3 AZ
- Order at Shard level
- You can merge shard / reshard
- Producers
- KPL for C++, Java
- Kinesis Agent: can write to Data Stream and Kinesis Data Firehose
- Consumers
- KCL: Checkpoint, coordinated reads write to DynamoDB
- You can put Source Records into S3
- You can put transformation failures job into S3
- You can put Delivery Failures job into S3
- FIrehose is near real time because it has buffer
- Max size for buffer is 32 MB, once reach that will flush buffer
- Max time is 1 min
- Buffer sizse can be automaticlly increase to increase throughput
- If you need real-time flush, use Data streams with Lambda
