Page Comparison

...

• Install the relevant cdap-hbase-compat package on all hbase nodes in your cluster in order to use the replication status coprocessors. Note that due to HBase limitations, these coprocessors cannot be used on hbase 0.96 or 0.98. Compat packages are:

  1. cdap-hbase-compat-1.0
  2. cdap-hbase-compat-1.0-cdh
  3. cdap-hbase-compat-1.0-cdh5.5.0
  4. cdap-hbase-compat-1.1
  5. cdap-hbase-compat-1.2-cdh5.7.0

• Modify hbase-site.xml on all hbase nodes to enable hbase replication, and to use the CDAP replication status coprocessors

  Code Block
  <property> <name>hbase.replication</name> <value>true</value> </property> <property> <name>hbase.coprocessor.regionserver.classes</name> <value>co.cask.cdap.data2.replication.LastReplicateTimeObserver</value> </property> <property> <name>hbase.coprocessor.wal.classes</name> <value>co.cask.cdap.data2.replication.LastWriteTimeObserver</value> </property>

• Modify hbase-env.sh on all hbase nodes to include the hbase coprocessor in the classpath 

  Code Block
  export HBASE_CLASSPATH="$HBASE_CLASSPATH:/opt/cdap/<hbase-compat-version>/coprocessor/*"   # for example, if you're on cdh5.5.x and have installed the cdap-hbase-compat-1.0-cdh5.5.0 package: export HBASE_CLASSPATH="$HBASE_CLASSPATH:/opt/cdap/hbase-compat-1.0-cdh5.5.0/coprocessor/*"

• Restart hbase master and regionservers

• Enable replication from master to slave 

  Code Block
  master hbase shell> add_peer '[slave-name]', '[slave-zookeeper-quorum]:/[slave-zk-node]'   example: master hbase shell> add_peer 'hbase2', 'hbase2-master.domain.net:2181:/hbase'

• Enable replication from slave to master 

  Code Block
  slave hbase shell> add_peer '[master-name]', '[master-zookeeper-quorum]:/[master-zk-node]'   example: slave hbase shell> add_peer 'hbase1', 'hbase1-master.domain.net:2181:/hbase'

• Check replication is working 

  Code Block

  master hbase shell> create 'repltest', 'f' slave hbase shell> create 'repltest', 'f' master hbase shell> enable_table_replication 'repltest' slave hbase shell> alter 'repltest', { 'NAME' => 'f', 'REPLICATION_SCOPE' => 1 } master hbase shell> put 'repltest', 'masterrow', 'f:v1', 'v1' slave hbase shell> put 'repltest', 'slaverow', 'f:v1', 'v1' master hbase shell> scan 'repltest' slave hbase shell> scan 'repltest'

...

Setup replication for the Kafka brokers you are using. Kafka MirrorMaker is (https://kafka.apache.org/documentation.html#basic_ops_mirror_maker and https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330) is the most common solution.

...

CDAP requires that you provide an extension that will perform HBase-related DDL operations on both clusters instead of just one. To create an extension, you must implement the HBaseDDLExecutor class:

...

To deploy your extension, run these steps on both your master and slave clusters:

1. Create an extension directory directory:

   Code Block
   $ mkdir -p /opt/cdap/master/ext/hbase/repl

2. Copy your jar JAR to the directory directory:

   Code Block
   $ cp myextension.jar /opt/cdap/master/ext/hbase/repl/

3. Modify cdap-site.xml to use your implementation of HBaseDDLExecutor HBaseDDLExecutor:

   Code Block
   <property> <name>hbase.ddlexecutor.extension.dir</name> <value>/opt/cdap/master/ext/hbase</value> </property>

4. Modify cdap-site.xml with any properties required by your executor. Anything prefixed by 'cdap.hbase.spi.hbase.' will be available through the Context object passed into your executor's initialize method:

   Code Block

   <property> <name>cdap.hbase.spi.hbase.zookeeper.quorum</name> <value>hbase-master-i18003-1000.dev.continuuity.net:2181/cdap</value> </property> <property> <name>cdap.hbase.spi.hbase.zookeeper.session.timeout</name> <value>60000</value> </property> <property> <name>cdap.hbase.spi.hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>cdap.hbase.spi.hbase.bulkload.staging.dir</name> <value>/tmp/hbase-staging</value> </property> <property> <name>cdap.hbase.spi.hbase.replication</name> <value>true</value> </property>

Before starting CDAP on the master cluster, run a command on the slave cluster to load the HBase coprocessors required by CDAP onto the slave's HDFS HDFS:

slave$ cdap setup coprocessors

...

Start CDAP on the master cluster:

master$ cdap master start

...