Build Pipelines

This document describes the CDAP build pipelines on Cask's build system. There are two main pipelines, the CDAP development pipeline, and the CDAP release pipeline. Both pipelines start with a single build, CDAP-DRC, which performs RAT and checkstyle checks on all branches in the CDAP repository. On success, the build triggers the child builds, which are based on the branch name.

Development Pipeline

Branches: (bugfix|feature)/CDAP.* and (bugfix|feature)_release/CDAP.*, or for UI: bugfix|feature)-ui/CDAP.* and (bugfix|feature)_release-ui/CDAP.*

The development pipeline is for pre-merge testing and will perform unit tests on the given branch. There are two builds which cover this: CDAP-DUT, which will match (bugfix|feature)/.* branches and merge with the develop branch before building, and CDAP-RUT, which will match any (bugfix|feature)_release/CDAP.* branches and merge with the latest release/.* branch of CDAP. If a release bugfix or feature branch needs to merge against a different parent release, the plan branch will need to be configured manually to do so.

Similarly for UI code branches, there are two UI-specific builds: CDAP-UDUT will match bugfix|feature)-ui/CDAP.* branches and merge with develop before building, and CDAP-URUT will match any (bugfix|feature)_release-ui/CDAP.* branches and merge with the latest release/.* branch of CDAP.  If a release bugfix or feature branch needs to merge against a different parent release, the plan branch will need to be configured manually to do so.

Release Pipeline

Branches: develop and release/.*

The release pipeline is for post-merge building and testing. It uses the cdap-build repository to include projects outside the main CDAP repository, such as Cask Tracker, which are shipped with CDAP releases. This is done in the CDAP-BUT build, which performs unit tests on the given branch, then builds all of the Maven artifacts for a CDAP release and deploys JARs to Sonatype OSS for public consumption. Snapshot JARs are available immediately. Release JARs are stored in a staging repository and must be released manually by Cask before they are accessible. After completion of the main artifacts build, CDAP-BUT will trigger CDAP-BA, an assembly build which creates secondary artifacts such as the SDK Docker Image and Virtual Machine Image, and IT-ITS, an integration test build using in-memory CDAP.

To release artifacts to the public, the CDAP-DA build will download them from CDAP-BUT and CDAP-BA and deploy them to the appropriate public repository. CDAP-DA requires artifacts from CDAP-BUT and CDAP-BA, but it is not triggered and must be run manually to deploy.