...
- Support both transactional and non-transactional message publishing
- Support both transactional and non-transactional message consumption
- Messages are ordered by publish time
- This helps address messages by timestamp
- Maintain the same message ordering for all consumers
- Consistent message ordering is the key to reprocess messages safely
- Easy to write tool to inspect what's inside the queue
- Each message has a unique message ID. The message IDs should be sortable as the same order as the message ordering as seen by the consumers
- Latency must be as low as possible.
- For messages published non-transactionally, they should be available for consumption right after it is persisted in the messaging system
- For messages published transactionally, they will be persisted to the message system as close as possible to the transaction commit time
- Use either client side buffering or auxilary payload table to minimize the "blocking" effect of non-committed transaction
- Consumers poll for new messages instead of pushing from the server
- This gives consumer more control
- Messages are immutable once they get "accepted" by the messaging system
- For non-transactional message, "accepted" means the message is persisted
- For transactional messages, "accepted" means the transaction of that message is committed
- Hence ordering are immutable as well
- A centralized messaging service will be provided and all messaging clients only interact with the messaging service through REST
- Multiple instances of the messaging service can be up and running at the same time for scalability
Message Table
The Message Table provides ordering of messages and entries in this table are immutable. Immutability is the key to provide consistent ordering across consumers. Also, immutable means entries can be cached for multiple consumers to consume without consulting the backing storage until the cache runs out. This provides an easy way to boost performance.
...
Transactional consumption basically follow the same procedures as the non-transactional one, with the addition that it will stop at the first uncommitted message when scanning the Message Table. The transaction information comes from the client and it is the client responsibility to open a new transaction in order to get a new snapshot of committed messages in the messaging system. The will increase the latency of message consumption, but the technique described above for message publishing, this latency should be minimal in the range of less than a second.
Message Retention
...
All published message will be stored in the messaging system until it expires, which is a property defined by topic. Since we use one Message Table for all topics, we cannot use the simple TTL mechanism as provide by HBase. The support of message retention can be broken down into two parts
- Message consumption
- During message consumption, we can simply apply a lower bound on the scan start row based on the TTL setting of the topic being consumed from.
- Message cleanup
- In local mode, LevelDB is used as the backing store. The messaging system will have a cleanup thread up and running and periodically scan and delete old entries based on the TTL setting of each topic.
- In distributed mode, since HBase is used as the backing store, the best way to do cleanup is to use a coprocessor that drops expired cells at flush/compaction time. The TTL setting can be made available to the coprocessor through the table attribute.
Implementation considerations
- To avoid scanning the HBase too much, the messaging service should cache recent scans in memory. This is based on the assumption that consumers shouldn't be falling too far behind, which is one of the reason why we want a messaging system to provide faster reaction time.
- Multiple instances of messaging service instances
Security
API
REST
Publish messages
...