Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Goals

 

Reference

 

Requirements

  • Support Active-Active and Active-Passive configuration
  • Provide tool or status on whether the replication is complete or is in a safe state
  • Support the ability to replicate HBase DDL to remote cluster – support creation of tables dynamically
  • Handle Kafka offset management across multiple clusters (Shortcoming of Mirror Maker)
  • Support replication of routing configuration stored in Zookeeper to remote cluster

Replications:

  1. HDFS:
    1. Hadoop Distcp is a tool used for large inter/intra-cluster copying. It uses MapReduce to effect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which will copy a partition of the files specified in the source list.

    2. Hadoop Distributed Copy Command: http://hadoop.apache.org/docs/r1.2.1/distcp2.html

    3. Cloudera Ditscp page: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_admin_distcp_data_cluster_migrate.html

    4. HortonWorks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_Sys_Admin_Guides/content/using_distcp.html


  2. HBase:
  3. Kafka:
  4. FileSets

 

 

Challenges

 

Open Questions

  • No labels