Introduction 

Phase 1 of replication is to support a hot-cold setup where CDAP data is replicated from one cluster to another using existing tools for replicating underlying infrastructure.

Goals

Allow manual failover from a hot cluster to a cold cluster.

User Stories 

• As a cluster administrator, I want to be able to configure CDAP so that all HBase tables created by CDAP are set up to replicate data to another cluster
• As a cluster administrator, I want to be able to manually stop CDAP in one cluster and start it in another cluster with the exact same state
• As a cluster administrator, I want to be able to have a way to know when it is safe to start the cold cluster after the hot one has been shut down

Design

CDAP stores state in several systems:

HDFS

• Transaction snapshots
• Artifacts (jars)
• Streams
• FileSet based datasets
• Program logs

HBase

• CDAP entity metadata (program specifications, schedules, run history, metrics, etc.)
• Table based datasets
• Kafka offsets for metrics and logs
• Flow queues
• Messaging system data

Kafka

• unprocessed metrics
• unsaved log messages

Hive

• Explorable CDAP datasets and their partitions

For phase 1, much of the responsiblity for data replication falls to the cluster administrator. It is assumed that replication of HDFS, Hive, and Kafka will be handled by the cluster administrator. HDFS is usually done through regularly scheduled distcp jobs, or by using some distro specific tools, such as Cloudera's Backup and Data Recovery (http://www.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_about.html). Kafka can be done using MirrorMaker. Hive can be done by replicating the data (HDFS and/or HBase), and by replication the metastore through whatever replication mechanisms are available to the relational DB behind backing the metastore. All of this can be setup outside of CDAP.

HBase, however, One thing CDAP needs to ensure is that there are no cluster specific values in any of the metadata. For example, the namenode should not be in any of the system metadata, otherwise things will fail when the data is replicated over to the slave and the slave is started.

HBase DDL Design

HBase DDL will require some hooks in CDAP, because replication must be setup for every table when it is created, and before any data is written to it. CDAP will define an interface to create, modify, and delete HBase tables.  By default, it will be implemented by the current code, which only creates tables in the local HBase instance.  Another implementation can be used by setting a property in cdap-site.xml that specifies the class to use. The jar containing the class must be included in the cdap classpath.  This custom class could, for example, make an http call to an external service to create the needed hbase tables.

Interface to be put here

Approach

Approach #1

Approach #2

API changes

New Programmatic APIs

New Java APIs introduced (both user facing and internal)

Deprecated Programmatic APIs

New REST APIs

Deprecated REST API

CLI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

UI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspectDesign details are at HBase DDL SPI.

Replication Status

Cluster administrators will require a way to tell when it is safe for a cold cluster to be started up. Design details are at Replication Status Tool. 

Kafka offset mismatches

MirrorMaker is not much more than a Kafka client that consumes from source topics and writes the same messages to some destination. As such, MirrorMaker does not offer any guarantees about a message from the source being written to the same partition and offset in the destination. The log saver and metrics processor store Kafka offsets per topic partition in HBase, and will need to be modified to handle a cluster failover. Design details are at Resolving Kafka Offset Mismatches.

API changes

New Programmatic APIs

See HBase DDL SPI

Deprecated Programmatic APIs

No programmatic APIs will be deprecated

New REST APIs

No new REST APIs will be added to the platform. There may be new REST APIs used by an external service used to handle table DDL and replication.

Deprecated REST API

No REST APIs will be deprecated

CLI Impact or Changes

• No Changes

UI Impact or Changes

• No Changes

Security Impact 

Cluster administrators are responsible for setting up replication, but we should understand what is required from a security perspective to replicate hdfs and hbase data.

Impact on Infrastructure Outages 

With replication, there is now another cluster that is required. If the cold cluster suffers an outage, replication will eventually catch up once service is restored. This assumes the outage lasts is for a shorter duration of time than how long HBase keeps events in its WAL.

Test Scenarios

Releases

Release 4.

1.

0 

Phase 1 work is scheduled for release 4.1.0.

Related Work

• Work #1
• Work #2
• Work #3

Future work