Page Comparison

...

Row Key Format

concat(topic, generation, publish_timestamp, seq_id)

• topic
  
  • String representation of the topic name
• generation
  • 4 bytes int representing the generation of this topic. Generation increase by one when at topic creation time.
• publish_timestamp
  
  • 8 bytes time in milliseconds
• seq_id
  
  • 2 bytes sequence ID. It is for uniquely identifying messages published in the same millisecond.
  • The 2 bytes gives us 64M messages per second per topic, which is well over the limit we need

...

Row Key Format

concat(topic, generation, transaction_write_pointer, write_timestamp, p_seq_id)

• topic
  • String representation of the topic name
• generation
  • 4 bytes int representing the generation of this topic. Generation increase by one when at topic creation time.
• transaction_write_pointer
  • 8 bytes transaction write pointer of the transaction that the message was published from
• write_timestamp
  • 8 bytes time in milliseconds of the time when the message was persisted
  • This is for ordering messages published in the same transaction
• p_seq_id
  • 2 bytes sequence ID. It is for uniquely identifying messages published in the same millisecond.

...

• Response
  404 NOT FOUND if topic is not present
  200 OK Response body contains an Avro binary or JSON object based on the request Content-Type with the response schema

Programmatic

Programmatic API will be used by clients to publish/poll messages from the Messaging Service. They are just a wrapper around the REST APIs.

• TopicId extends NamespaceId with a string that represents the topic name.

Creating Topic:

void create(TopicId topic) throws TopicAlreadyExistsException;

Updating Topic properties:

void update(TopicId topic, Map<String, String> properties) throws TopicAlreadyExistsException;

Deleting Topic:

void delete(TopicId topic) throws TopicNotFoundException;

Publishing Message:

For publishing messages non-transactionally:

• void publish(TopicId topic, List<byte[]> messages) throws TopicNotFoundException;
• void publish(TopicId topic, byte[] message) throws TopicNotFoundException; 

For storing messages in the PayloadTable (requires Transaction)

• void store(TopicId topic, byte[] message, Transaction transaction) throws TopicNotFoundException;
• void store(TopicId topic, List<byte[]> messages, Transaction transaction) throws TopicNotFoundException;  

For publishing messages without using payload table or to publish messages stored earlier in the PayloadTable:

• void publish(TopicId topic, Transaction transaction) throws TopicNotFoundException;
• void publish(TopicId topic, byte[] message, Transaction transaction) throws TopicNotFoundException;
• void publish(TopcId topic, List<byte[]> messages, Transaction transaction) throws TopicNotFoundException;

For rolling back messages that were published transactionally:

• void rollback(TopicId topic, Transaction transaction) throws TopicNotFoundException;

Consuming Message:

For consuming messages non-transactionally:

• List<Message> poll(TopicId topic) throws TopicNotFoundException; 
• List<Message> poll(TopicId topic, long timestampInMs, boolean inclusive) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, MessageId messageId, boolean inclusive) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, long timestampInMs, boolean inclusive, int maxMessages) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, MessageId, messageId, boolean inclusive, int maxMessages) throws TopicNotFoundException;
   

For consuming messages transactionally (timestamp used for polling will correspond to the publish timestamp):

• List<Message> poll(TopicId topic, Transaction transaction) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, MessageId id, boolean inclusive, Transaction transaction) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, long timestampInMs, boolean inclusive, Transaction transaction) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, MessageId id, boolean inclusive, int maxMessages, Transaction transaction) throws TopicNotFoundException;
• List<Message> poll(TopicId topic, long timestampInMs, boolean inclusive, int maxMessages, Transaction transaction) throws TopicNotFoundException;

Public Programmatic API

Consuming Message (when executed within a Transactional, then messages are consumed transactionally, but if it is used outside of a Transactional then messages are consumed non-transactionally) :

...

Data Cleanup In Tables

Various cleanup of data needs to be performed. And the clean up strategy varies based on the underlying storage - LevelDB vs HBase. The matrix below describes the combinations and types of cleanup we need to do and how it is going to be performed.