Introduction 

Today, every property a provisioner supports is exposed to users when creating a profile. This is because all existing properties can vary from profile to profile. However, there are certain types of properties that should not be exposed to users and should be shared across all profiles that use the provisioner. These types of settings usually need to be set once and do not change frequently.

Goals

Provide a way for CDAP administrators to configure provisioner internals that should not be exposed to end users.

Use Cases

• An administrator wants to ensure that every cluster created by CDAP is labeled with the name of the organization, and the CDAP environment (dev, staging, or prod). This will help organize clusters, as well as track costs for clusters created by CDAP.
• An administrator installs CDAP in their cloud environment. The clusters created by CDAP always live in the same cloud environment as CDAP and in the same network. The administrator wants to configure the provisioners to use the internal IP addresses of those clusters when SSHing instead of the external IP addresses. 

User Stories 

• As a provisioner developer, I want to define a system property that is common across all profiles that use the provisioner
• As a provisioner developer, I want don't want system properties to change while CDAP is running
• As a system administrator, I want to be able to configure the system properties of my provisioners
• As a system administrator, I don't want CDAP users to be able to view or modify provisioner system properties

Design

The Provisioner interface will be modified to have an initialize method that takes a map of properties. Initialize will be called once when the provisioner is created, and is guaranteed to be called before any other method.

public interface Provisioner {
  ...

  /**
   * Initialize a provisioner with its system properties.
   * This is called once before any other methods are called.
   */
  void initialize(Map<String, String> systemProperties);
}

This can be used to implement the use cases mentioned earlier in this design.

public class DataprocProvisioner {
  private Set<String> labels;
  private boolean useInternalIP;


  void initialize(Map<String, String> systemProperties) {
    labels = new HashSet<>();
    String systemLabels = systemProperties.get("labels");
    if (systemLabels != null) {
      for (String systemLabel : systemLabels.split(',')) {
        labels.add(systemLabel);
      }
    }
    useInternalIP = Boolean.valueOf(systemProperties.get("use.internal.ip");
  }


  ...
}

Provisioner system properties are specified in the CDAP configuration file. When instantiating a Provisioner, CDAP will look for any properties in the CConfiguration that are prefixed with 'provisioner.system.properties.<provisioner-name>.'. Anything after the prefix will be used as a property key and the corresponding value will be the property value. For example, if the provisioner above is named 'gcp-dataproc' and cdap-site.xml contains:

  <property>
    <name>provisioner.system.properties.gcp-dataproc.labels</name>
    <value>prod,finance</value>
  </property>


  <property>
    <name>provisioner.system.properties.gcp-dataproc.use.internal.ip</name>
    <value>true</value>
  </property>

then the 'systemProperties' map in the above code example would have two values: 'labels' = 'prod,finance' and 'use.internal.ip' = 'true'.

API changes

New Programmatic APIs

Provisioner gets a new initialize method:

public interface Provisioner {
  ...
  void initialize(Map<String, String> systemProperties);
}

Deprecated Programmatic APIs

None

New REST APIs

None

Deprecated REST API

None

CLI Impact or Changes

None

UI Impact or Changes

None

Security Impact 

None

Impact on Infrastructure Outages 

None

Test Scenarios

Releases

Release 5.1.0

Future work

None planned