Everything about RabbitMQ
RabbitMQ vs Apache Kafka
資料結構差異
The Queue(RabbitMQ) vs. The Log (Apache Kafka)
-
Queue
- FIFO data structure
- Optimized for efficient write and read operations from either end of the sequence
- Stored data is transient in nature
- Not shared between applications
-
Log
-
Also known as write-ahead logs, transaction logs and commit logs
-
Append-only. Ordered by time
-
Unique log sequence numbers (可以用來識別這個 log 在 log data 的 position)
-
Stored data is persistent in nature
-
Shared between applications (多個 Reader 讀完 data 後，不刪除 data，不影響其他Reader)
-
RabbitMQ - Building Blocks
-
Routing
- Fanout Exchange
- Direct Exchange
- Topic Exchange
- Header Exchange
- Consistent Hashing Exchange
- Random Exchange
- Sharding
-
Queues and Consumers
- Queues
- Competing Consumers
- Non-Competing Consumers
- Ephemeral Queues
- Lazy Queues
- Priority Queues
-
More
- Deadletter Exchange
- Alternate Exchange
- Virtual Hosts
Traditional queues on steroids
Exchanges, queues and bindings
- Publisher send messages to Exchanges
- Exchanges route messages to Queues and even other Exchanges
- Consumers read from Queues
- Bindings link Exchanges to Queues and even Exchanges to Exchanges
[Publisher]-->[Exchange] ──binding-->[Queue]-->[Consumer]
Publish - Subscribe
Fanout Exchange
-
Simplest Publish-Sbuscribe pattern
-
Each consumer receives every message
Publish - Subscribe with Routing
Direct Exchange
-
Routing by Routing Key
-
Exact match routing key = binding key
Topic Exchange
-
Wildcard routing by Routing Key
-
*= match 1 word -
#= match multiple words
Header Exchange
-
Routing by message headers
-
Note: Quite slow, needs a lot of CPU, does not scale very well
Point-to-Point Messaging
Default Exchange
-
Direct exchange type
-
All queues have implicit binding to the default exchange
-
Route messages by Routing Key to Queue Name
Protecting Against Routing Failures
Alternate Exchange
-
Not an exchange type, but a configured relationship between two exchanges
-
Route unrouteable messages to an alternate exchange
Scaling Out Consumers
Non-Competing Consumers
-
Each consumer receives every message
Scale Out with Competing Consumers
-
Each Shipping consumer receives a subset of the messages
Scaling Out Queues
Random Exchange
-
Load balance messages across queues randomly
Scaling Out Queues with Data Locality
Consistent Hashing Exchange
- Partition messages across queues via hashing function over routing key, message header or message property
[9:17 image]
Sharding Plugin and Modulus Exchange
- Partition messages across queues over multiple hosts via hasing function on the routing key.
[10:13]
Queue Limits and Deadletter Exchanges
Apply limits to queues
- Length
- Size
- Time limits (TTL)
Eject messages from a queue when:
- A queue size/length limit reached
- A message has spent longer than the TTL time limit in the queue
Route to a deadletter exchange to avoid message loss
[10:15] from queue to exchange then another queue
Ephemeral Queue
-
Auto-delete queue
如果最後一位消費者取消訂閱queue或者connection failed
-
Queue Expiration (TTL)
如果過了五秒鐘還沒有任何consumer，就自動解構(destruct)
-
Exclusive Queues
消費者可以定義它們自己的私有queue，其它consumer無法連接上
-
Auto-Delete Exchanges
Lazy Queues
- Memory optimized queues
Priority Queues
- No longer FIFO
- Publisher sets priority on messages
- Priority Queue moves higher priority messages ahead of lower
- Drawbacks - blocked low priority messages, low priority can eject high priority when queue is full
Virtual Hosts
- Allows Multi-Tenant architecture
- Isolation of groups of exchanges, queues and users
- No routing between two virtual hosts