Namespace Migration

1. If namespaces are to be migrated from one cluster to another cluster, both source and destination clusters should be running the CDAP with the same version.

2. Both clusters need to have the same timestamp, otherwise the information such as TTL for the stream will be invalidated.

3. All programs on the old CDAP cluster are stopped and no additional activity (such as ingesting the data to the stream) is happening on the old cluster.

4. There are no pending events in the flow queue before the migration process starts. It will be nice to have the support from the platform for draining the queue data.

5. We will not migrate any transient data such as stream state store and queue states. Migration will only be started once the existing data is processed.
   

Scope for 3.3:

In 3.3 we will focus on the migration of the datasets and streams associated with the applications.

Migration of the application consists of the following steps:

1. Ensure that there is no unprocessed data in the application. This will be manual process for 3.3.

2. Migrate the datasets and streams associated with the application.

3. Deploy the application in the new namespace.

4. We will not migrate the operational data such as metrics and logs as a part of 3.3.

Following are some possible use cases for the migrating the application datasets/streams:

1. Migrate the dataset X from namespace ‘NS1’ to namespace ‘NS2’. [NS1.X -> NS2.X]

2. Migrate the dataset X from ‘NS1’ to ‘NS2’ updating its name to Y. [NS1.X -> NS2.Y]

3. Make the copy of the dataset X in ‘NS1’. [NS1.X -> NS1.Y]

Once the application data is migrated, application will need to be deployed manually. If there are any changes in the dataset names, corresponding changes in the application would require. Migration of the application forms a baseline case for migrating the namespaces.

Migration of the HDFS directories:  

CDAP stores FileSets and Streams in HDFS. We can figure out the filesets and the streams used in the namespace and can use distcp to migrate the corresponding HDFS files to the new cluster. FileSets also support reading from the files which are managed by the external process. This allows accessing the existing locations outside of the CDAP. We will need to migrate those files as well, however doing so may cause conflicts if the file location already exists on the target cluster.

Migrating the user datasets:  

 User datasets (except FileSets) are stored as HBase tables. However we cannot directly copy those tables over to the target cluster. Every HBase cell can have multiple versions marked by the transaction-ids. These transaction-ids generated by the Transaction Manager on the old cluster may conflict with the transaction-ids generated on the new cluster. For e.g. Consider tx1 is the latest transaction id generated for the table on the old cluster. If we simply copy the table, then the copied table will also have transaction id as tx1. However the new cluster might have generated the transaction id tx1 and its invalidated. In this case, the copied data from the new table will not be visible on the new cluster.

Potential solutions:

1. 1. Check if it is possible to inject CDAP HBase implementation in the CopyTable utility.

   2. Write the custom MapReduce to copy the HBase user datasets. This MapReduce program will get the Transaction-Id from the Transaction Manager running on the new cluster.

Solving namespace migration use cases with the application migration scheme (NOT FOR 3.3):

1. Migrate data from namespace ‘NS1’ on the old cluster to the namespace ‘NS2’ on the new cluster. ‘NS2’ can be on the same cluster or on the new cluster.

    for each app : 'NS1'.applications
       for each dataset : app.datasets
          migrate the dataset either using distcp (if it is filesets) or using hbase table copy operation
     for each stream : app.streams
        migrate the stream files using distcp   

   
   

2. Merging of the namespaces ‘NS1’ and ‘NS2’ into namespace ‘NS3’. Namespace ‘NS3’ can be on the same cluster or on the new cluster.

    for each namespace 'NS' : Namespaces to merge
       for each app : 'NS'.applications-to-migrate
          for each dataset : app.datasets
             migrate the dataset either using distcp (if it is filesets) or using hbase table copy operation
        for each stream : app.streams
           migrate the stream files using distcp   

3. Splitting of the namespace ‘NS1’ into multiple namespaces say ‘NS2’ and ‘NS3’. Namespaces ‘NS2’ and ‘NS3’ can exist on the existing cluster or on the new cluster.

   // TBD