Goals

CDAP and CDAP Applications have the ability to withstand short and transient infrastructural outages
During interruption of underlying services (one or more), CDAP or CDAP Applications can operate under degraded performance/limited functionalities
- Users will not be able to perform operations like deploying apps, starting programs or new data or application lifecycle operations.
  - However all the applications that are running, should be running
Once interruption in the underlying service is resolved or services come back to normal operation, the CDAP and CDAP Application will go back to normal state
Interruptions in service would be due to node failure, service failures or compatible rolling upgrades or downgrades in progress
Does not include in-compatible upgrades or downgrades of underlying infrastructure
Does not include long unavailability of service and infrastructure

Resiliency Scenarios

Maintenance of underlying infrastructure
- Administrators
Compatible Upgrade or Downgrade of underlying infrastructure Hadoop components
- Underlying Hadoop infrastructure is either being upgraded or downgraded and the expectation is that CDAP and CDAP Applications would tolerate and be resilient to infrastructure services not being available during the process. The process could anywhere between 30 mins - 18 hours or more.
- During this period the CDAP and CDAP Applications can operate in degraded mode.
- The Hadoop infrastructure upgrade has to be compatible with CDAP and CDAP Application in order to have smooth upgrades
- In case, there are issues during the upgrade, CDAP should also be resilient to rollbacks
- CDAP and CDAP Applications should also be able to withstand compatible downgrades
- The compatibility matrix should be available to users to ensure smooth upgrades
Upgrade / Downgrade of CDAP
- Upgrade the version of CDAP. Major and Minor version could have different impacts. We will discuss about these impact further in the document.
- Rolling back the upgrade process
- CDAP version compatible matrix available to users
Upgrade / Downgrade of CDAP Applications

The underlying Hadoop Infrastructure can be subjected to upgrade or downgrade anytime. The

Use cases

User wants to upgrade the underlying Hadoop distribution (CDH / HDP / MapR / BigTop) within a major release
User wants to perform rolling upgrade of the underlying infrastructure
User

Define long and short/transient outages
- More information to gathered here to understand the length of outages.
- When outages are multiple hours, how should the system handle these.

Send supported HBase version by CDAP
Gather information about CDH version compatibility changes – Talk to Cloudera and compile

[3.6] CDAP Service version and upgrade support
[3.6] Application versioning
[4.0] Messaging Service with goal of centralizing all transactional activities for metadata in HBase
~~[4.0?] Non-Transactional datasets~~
[4.0?] HBase Coprocessor Upgrade Management — Handling minor version changes efficiently without disabling HBase Tables.
[4.0?] Upgrade tool improvements — Coprocessor Upgrade removal, faster data conversions if needed, smarts to reduce the impact to running services
[4.0?] CDAP Service Upgrade capability, might have Apache Twill change
[4.0?] Move configuration and operational updates to messaging services