Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

This document lists detailed steps required for setting up Replication for CDAP:

Cluster Setup

CDAP replication relies on the cluster administrator setting up replication on HBase, HDFS, Hive, and Kafka. It is assumed that CDAP is only running on the master cluster. It is assumed that you have not started CDAP before any of these steps.

HBase

  • Install the relevant cdap-hbase-compat package on all hbase nodes in your cluster. Compat packages are:

    1. cdap-hbase-compat-0.96
    2. cdap-hbase-compat-0.98
    3. cdap-hbase-compat-1.0
    4. cdap-hbase-compat-1.0-cdh
    5. cdap-hbase-compat-1.0-cdh5.5.0
    6. cdap-hbase-compat-1.1
    7. cdap-hbase-compat-1.2-cdh5.7.0

  • Modify hbase-site.xml on all hbase nodes to enable hbase replication, and to use the CDAP replication status coprocessors

    <property>
  <name>hbase.replication</name>
  <value>true</value>
</property>
<property>
  <name>hbase.coprocessor.regionserver.classes</name>
  <value>co.cask.cdap.data2.replication.LastReplicateTimeObserver</value>
</property>
<property>
  <name>hbase.coprocessor.wal.classes</name>
  <value>co.cask.cdap.data2.replication.LastWriteTimeObserver</value>
</property>

  • Modify hbase-env.sh on all hbase nodes to include the hbase coprocessor in the classpath 

    export HBASE_CLASSPATH="$HBASE_CLASSPATH:/opt/cdap/<hbase-compat-version>/coprocessor/*"
 
for example, if you're on cdh5.5.x and have installed the cdap-hbase-compat-1.0-cdh5.5.0 package:
export HBASE_CLASSPATH="$HBASE_CLASSPATH:/opt/cdap/hbase-compat-1.0-cdh5.5.0/coprocessor/*"
  • Restart hbase master and regionservers

HDFS

 

Hive

Kafka

 

CDAP Setup

 

  1. Setup hbase-site.xml on ALL nodes:

    1. These properties need to be added. Choose Coprocessors based on the HBase version running on the cluster. In this example, its HBase 1.0. 

       <property>
 <name>hbase.replication</name>
 <value>true</value>
 </property>
 <property>
 <name>hbase.coprocessor.regionserver.classes</name>
 <value>co.cask.cdap.data2.replication.hbase10.LastReplicateTimeObserver</value>
 </property>
 <property>
 <name>hbase.coprocessor.wal.classes</name>
 <value>co.cask.cdap.data2.replication.hbase10.LastWriteTimeObserver</value>
 </property>
  2. The following steps are required for a secure cluster:
    1. Set up JAAS Configuration.
    2. Sync /etc/hosts on all the nodes
  3. SPI Setup:
    1. <>
  4. Restart HBase Master and Region Servers. 
  5. Setup HDFS Replication:
    1. This can be achieved by using hadoop tool distcp with regular intervals. For help on distcp check here. 
    2. For secure clusters, distcp can be run with cdap keytab. 
  6. Start CDAP on Master Cluster.
  7. Verify cdap Tables are created on both Master and Slave clusters. 

  • No labels