Page Comparison

...

• To avoid scanning the HBase too much, the messaging service should cache recent scans in memory. This is based on the assumption that consumers shouldn't be falling too far behind, which is one of the reason why we want a messaging system to provide faster reaction time.
• Multiple instances of messaging service instances (not part of the initial scope)
  • When we reached the point where we need to scale the messaging service to multiple instances, there are couple choices on how it looks like
    1. All instances are equal. This mean all instances can handle publishing and consumption from all topics
       1. Different instances need to generate different seq_id for the same timestamp to avoid key collision. We potentially can use the instance ID as the first byte in the sequence id.
       1. (Pro) Easy to implement as there is no coordination needed
       2. (Con) The consumer scan cache can take up a lot of spaces because all instances would be caching for all topics
    2. Perform resource assignment through the ResourceCoordinator such that a given topic would only be handled by a sub-set of instances only
       1. Still need to handle sequence id generation to avoid key collision
       2. (Pro) Most flexible to have a trade between scalablity vs resource usage (memory cache) and can be tuned per topic based on traffic
       3. (Con) Implementation can be complicated as it involves
          1. Usage of ResourceCoordinator
          2. Usage of DistributedLock for the topic transition to have a guaranteed leader set for a topic
          3. Traffic forwarding / retry needed during topic transition from one instance to another

Security

TBA

API

REST

Publish messagesTBA

Programmatic

TBA