Introduction 

One of the reasons CDAP must be stopped before an upgrade is so that the upgrade tool can be run to update the coprocessors for all CDAP tables. In order to minimize downtime, we would like to be able to upgrade coprocessors in a rolling fashion.

Goals

Design a method to upgrade CDAP HBase coprocessors in a rolling fashion, with minimal downtime.

User Stories 

• As a cluster administrator, I want to be able to upgrade CDAP coprocessors without stopping CDAP
• As a cluster administrator, I want to be able to upgrade HBase without stopping CDAP

Design

Prior to 4.1.0, the way coprocessors are handled is that they are built and loaded onto hdfs when the dataset is created. When the HBase Table is created, it is configured with the hdfs path of the coprocessor(s), the classname of the coprocessor(s), and the priority. During a CDAP upgrade, CDAP is stopped, and an upgrade tool is run that loops through all tables, disables the table, builds and uploads the new coprocessor jars, modifies the table to point to the new coprocessor(s) on hdfs, then re-enables the table. This is nice in that CDAP manages coprocessors itself and cluster administrators don't need to know anything about coprocessors. It is not ideal in that it requires downtime in order to upgrade the coprocessor. 

Approach

Approach #1

The first approach adds a configuration setting 'master.manage.coprocessors' that defaults to 'true'. When true, CDAP handles coprocessors the same as before and cluster administrators don't have to do any additional work. However, it also means there will be downtime when upgrading CDAP or HBase. When set to false, when CDAP creates HBase Table, it will only specify the coprocessor(s) classname and priority and not the hdfs path. Instead of building coprocessor jars and placing them on hdfs, the CDAP coprocessor jars must be installed on every HBase node and included in the HBase classpath. 

In order to upgrade the coprocessors in a rolling fashion, cluster administrators must install the new CDAP coprocessor on the node to be upgraded and restart the regionserver. 

The downside to this approach is that it involves considerably more expertise and involvement from the cluster admin. This may also not work in Ambari or Clouder Manager. Based on our interactions with the Phoenix team and checking the HBase user group, this is how most people do rolling coprocessor upgrades.

Approach #2

Ideally, there would be a way to keep coprocessor management within CDAP and keep the jar on hdfs. For example, suppose each the coprocessor were basically just a wrapper that looks up the CDAP version in some predetermined table and uses that version to download the correct coprocessor jar on hdfs and build a classloader from it. With such a setup, upgrading CDAP would amount to loading the correct jar onto hdfs, updating the CDAP version in the table, then restarting HBase in a rolling fashion. However, this would not help with HBase upgrades, as an HBase upgrade it would require the wrapper coprocessor to be updated.

API changes

No changes to programmatic APIs

New REST APIs

No REST API changes

CLI Impact or Changes

• None

UI Impact or Changes

• None

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect

Test Scenarios

Releases

Release 4.1.0

Related Work

• Work #1
• Work #2
• Work #3

Future work