Build System & CI

Build Overview

We use Maven as our build system
- Docs: Available Plugins
- Docs: Help Plugin
The top level module to check out which orchestrates all the build process
- https://github.com/caskdata/cdap

How To Build CDAP Artifacts

git clone cdap project

git clone git@github.com:caskdata/cdap.git

Common Maven commands
- Clean all modules:
  - ```
  mvn clean
```
- Build all modules:
  - ```
  mvn package
```
- Run all tests:
  - ```
  mvn test
```
- Run checkstyle, skipping tests:
  - ```
  mvn package -DskipTests
```
- Build a particular module with all dependencies:
  - ```
  mvn package -pl [module] -am
```
- Run selected test:
  - ```
  mvn -Dtest=TestClass,TestMore*Class,TestClassMethod#methodName -DfailIfNoTests=false test
```
- See http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html for more
- Show dependency tree:
  - ```
  mvn package dependency:tree -DskipTests
```
- Show dependency tree for a particular module:
  - ```
  mvn package dependency:tree -DskipTests -pl [module] -am
```
- Show test output to stdout:
  - ```
  mvn -Dsurefire.redirectTestOutputToFile=false ...
```
- Offline mode:
  - ```
  mvn -o ...
```
- Change version:
  - mvn versions:set -DnewVersion=[new_version] -DgenerateBackupPoms=false

- List of projects, build order and the platform details like system properties and environment variables:
  - mvn help:system

Module Organization

The CDAP project is built as a multi-module maven project. While all modules are aggregated in the top-level CDAP pom, only the cdap-api module inherited from CDAP, while all others inherited from the parent module. By doing so, we have the flexibility on internal modules, while keeping the cdap-api simple.

How To Add A New Module

In situations when you want to add new module, create a sub-directory with the name of the new module. Then create a pom.xml inside that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>co.cask</groupId>
    <artifactId>parent</artifactId>
    <version>[Substitute correct version here]</version>
    <relativePath>../parent/pom.xml</relativePath>
  </parent>
  <artifactId>[new_module_name]</artifactId>
  <name>CDAP [new_module_name]</name>
  <packaging>jar</packaging>
  ....
</project>

Then modify the top level pom.xml to include the new module

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
  ....
  <modules>
    <module>cask-api</module>
    <module>cask-common</module>
    ...
    <module>[new_module_name]</module>
  </modules>
  ...
</project>

You can also easily do it using Intellij. Just right click on the top level "CDAP" project and select New->Module and follow the screen.

How To Create Debian Packages

To be completed.

Working With Our Nexus Repositories

We use Nexus Professional as our repository manager. It is available at https://repository.continuuity.com.

Our Repositories:

Our hosted repositories:

Repository Name	Type	Public/Private	Description
Snapshots	Hosted	Private	Where our snapshot build artifacts are stored
Releases	Hosted	Public	This is our public release repository, where our tested api jars and singlenode distribution are made available to the public
Releases-Internal	Hosted	Private	This is where our tested internal releases (cloud distribution, non-api jars) are made available (authentication required)
3rd Party	Hosted	Public	This is where we can put any custom third-party artifacts that are not available in Maven Central (for example, our patched version of hbase, latest kryo, etc); note this is a public repo

Additionally, we proxy many public repos:

Repository Name	Type	Public/Private	Description
Central	Proxy	Public	Maven Central
Apache Snapshots	Proxy	Public	Apache
Codehaus	Proxy	Public	Codehaus
Typesafe Releases	Proxy	Public	Typesafe - we use it for akka

For convenience, we have defined several repository Groups:

Repository Group Name	Type	Public/Private	Description
Smartifacts	Group	Private	This contains everything required for gradle to build CDAP: (Snapshots, 3rd party, central, typesafe)
Release-Candidates	Group	Private	Release builds publish artifacts here (public or private), where they are available as part of the build promotion process
Public Repositories	Group	Public	This contains everything publicly accessible (anonymously): Releases, 3rd Party, Central, Typesafe

Access Permission Levels:

Anonymous: Can access all "public" repos as noted above.

Nexus-users (crowd): users have read-only access to everything. All Cask engineers should be able to login via the Nexus UI.

Nexus-administrators (crowd): administrative users with full privileges. Any Cask engineer that needs it.

Gradle: gradle user is password protected and has deploy (update) permissions.