Goals

Fundamental need that is driving this work is the need to have multiple versions of the same program (in this case the driving factor is Service) running so that the requests are still served while the earlier version is shutdown etc. This is required for zero downtime of the service and is required when apps are updated.

User Stories 

• Salesforce is running v1.0.0 of journey builder application

• The application has a user service that receives events

• Salesforce wants to upgrade to v2.0.0 of journey building application with minimal down-time

• Salesforce also wants capabilities to send percentage of traffic of v2.0.0 version of the application before directing 100% traffic to it

• Salesforce wants to rollback to v1.0.0 with minimal downtime

Design

The above requirement suggests the need for application versioning. That is, the same application (identified by its name aka app name) can have multiple versions. Once we have that, if we have a service HTTP endpoint - /v3/namespaces/<ns-id>/apps/<app-id>/services/<service-id>/methods/<method-name>, this endpoint can be used continuously while underneath the user can now deploy an upgraded version of the app and the users of the endpoints are oblivious about that change and still served by one of the service versions. And the old service can then be stopped and this whole process doesn't affect the uptime of the endpoint.

We want to introduce application versioning without breaking backward compatibility. And current scope of the work involves only looking at handling Services. MR/Workflow/Spark already support concurrent runs and thus concurrent runs of these across multiple versions of the app shouldn't be an issue. However, what should happen when we have multiple versions of Flows run etc is not fully understood and thus we will not change the current design choice of not allowing concurrent runs of Flows (even across multiple versions of the App).

Approach

Application versions are chosen and set while creating the app by the user. 'Version' is represented as a string. If users don't provide a version, then the default version ("-SNAPSHOT") is used. If the app is created from an artifact, then the artifact version from the AppRequest is used as the version of the Application. Once created, an app version cannot be changed, unless the version ends with the string "-SNAPSHOT". So if a user is not using versioning at all, the current behavior of updating an app will work fine (since by default -SNAPSHOT is the version). For versioned endpoints (/apps/app-id/versions/version-id/services/MyService/start), the corresponding version of the app is used. If the user is using the non-versioned API (/apps/app-id/services/MyService/start), we will check if there are more than one version and if there is, then we will return an error code. This is so that backward compatibility (pre-app-version era and users who don't want to use app versions) is maintained. The only exception to this rule is the service method routing endpoints, which will use the configuration set to route the service requests appropriately by the CDAP Router (more details about that in the next section).

Reasoning: 

In CDAP, relationship between application and artifact are not 1-to-1. Multiple applications can be created from the same artifact by providing different application configuration, hence using the artifact version as the application version doesn’t work very well. E.g. in Hydrator, multiple pipelines (pipeline == application) are created from the same artifact, the hydrator artifact. We are introducing new REST endpoint for deploying application so that version is provided explicitly. I think we can have that endpoint default to use the artifact version as the application if the app version is not provided explicitly and I believe that should fit your use case pretty well. About the default version “-SNAPSHOT”, it is mainly for backward compatibility purpose. That is, if one deploy an artifact+app using the existing endpoint, we use “-SNAPSHOT” as the version internally so that it can be overwritten on redeploy (non SNAPSHOT application versions are immutable).

Service Routing

Routing to services with multiple versions running concurrently will have an additional feature of controlling the distribution of the requests. Instead of being completely random, the user can choose to allow, say 80% of requests to be served by "2.0.0" version, 10% by "1.3.1" and 10% by "1.3.0". This will be made by possible by allowing the user to set a distribution strategy for a particular service (ns-name, app-name, service-name) and it will be used by CDAP Router when deciding the service instance to forward the request to. If the route configuration is missing, then the request will use the default behavior configured in CConf. It could be random, min, max (string comparison) or just fail to route.

Support for other Program Types (other than Services)

We support concurrent runs for Spark, MapReduce and Workflow and thus none of their functionality needs any changes/updates. For Flows and Workers, we don't support concurrent runs today and we will retain the same logic across all versions of the app. So say a worker named MyWorker of app ver1 is running and if the user tries to start the same worker of app ver 2, we will return back an error with CONFLICT.

API Changes

REST API changes

CLI Changes

Authorization

We convert the CDAP's entity id to the ID used by Sentry. For simplicity, I propose that we do authorization simply at the application name level and not any more fine grained. So a user who has access X to that app, will get access X to all the versions of that app. This conversion can be done in the cdap-security-extn where we can check the permissions at that level?

Logs

We will have an additional logging systemTag, for application version id. With the new LogViewer, we don't explicitly provide the version number/program name etc but it can be returned as part of the JSON for that endpoint.

Metrics

We will have an appVersion tag for Metrics. So users can then query for metrics across all runs of a particular version of the program. Metrics for an app will not be deleted until all the versions of the app is deleted which is current behavior today. 

Preferences

No changes to Preferences. The hierarchy will be namespace -> app id -> program. There will be no app version dimension. Since we have runtime args, the users can use that if required.

Lineage
Audit
Metadata

TBD

Upgrade Step

Since the version used by default is "-SNAPSHOT", we will need to update the key in the Store (keys that start with appMeta) and add this default version to all the apps created. No other changes should be required.

UI Changes

We don't need any changes in the UI for 3.6.0. But for 4.0, we should display the application version along with artifact version when we show a specific Application. That is, artifact version should be displayed along with application version wherever the latter is planned to be shown.

Test Scenarios

Releases

Release 3.6.0 (Drop for 9/20)

• (Internal) Change ApplicationId to contain application version (will be part of the key of Store)
• Introduce REST API to create/delete Apps with non-default versions
• Introduce REST API to start/stop programs with versions
• Upgrade step to add default version to the keys of the Store 
• Check to make sure non-versioned API fail if multiple versions are present (since we need to define some behavior, we might as well implement the check) - Work will be involved in adding it to all endpoints
• Versions related endpoint - listing versions, getting app spec for a particular version (artifact id and app config etc)
• Service endpoint routing - ability to upload a config to provide a forwarding distribution logic at the Router
• Service endpoint routing when config is not present
   

Future Work

In scope for 4.0

• CLI
• Logging and Metrics support
• Metadata
• Lineage
• Audit
• Authorization (not required)