Approach #1 Data-prep app config to specify supported connection types and default connections (Preferred)

Administrators can specify a data-prep config JSON specific to their environment.

The config JSON can be used to specify a list of ConnectionTypeConfig 

• ConnectionTypeConfig can be used to specify that a connection needs to be created by default for the connection type and the properties for that connection.

• ConnectionTypeConfig can be used to specify the  set of connection types that are supported in DataPrep

{
  "disabledTypes": [ "file" ],
  "connections": [
    {
      "name": "gcsDefault",
      "type": "gcs",
      "properties": {
        "serviceFilePath": "~/gcp/service-config.json",
        "project": "cdap"
      }
    }, 
    {
      "name": "bqDefault",
      "type": "bigquery",
      "properties": { }
    },
    ...
  ]
}

Any connections specified in the config are created on start up of the Wrangler service. They cannot be deleted.

Example explanation :

• The 'file' connector type is in disabled connection type list, so no file connections are allowed.
• There is a default GCS connection that uses pre-defined credentials and project information as specified in the properties map. 
• There is a default BigQuery connection without any properties. The BigQuery connection will read required credentials from the environment.
• Kafka, Database and S3 connection types are configured to be visible, but no default connections are configured for them.

Surfacing the data-prep-config JSON to data-prep application

Data-prep config JSON can be specified through cdap-site.xml property

Option - 1

<property>
<name>data.prep.config.json</name>
<value></value>
<description>
   data prep config JSON content, this config is used to specify the connection-types that are   
   available as default and the connection-types that have to be disabled in data-prep.
</description>
</property>

Example 

<property>
<name>data.prep.config.json</name>
<value>
{
  "disabledTypes": [ "file" ],
  "connections": [
    {
      "name": "gcsDefault",
      "type": "gcs",
      "properties": {
        "serviceFilePath": "~/gcp/service-config.json",
        "project": "cdap"
      }
    }, 
    {
      "name": "bqDefault",
      "type": "bigquery",
      "properties": { }
    },
    ...
  ]
}
</value>
</property>

Notes

• We have to provide JSON content instead of JSON file path, as the CDAP UI service could be running on different instance than CDAP backend service

• Currently data-prep application is deployed and run separately on each namespace, however in the future when data-prep is moved to system namespace, it will be easier to have the data-prep application configured and deployed using bootstrap config files during CDAP startup instead of specifying through a cdap-site.xml property.

• By providing the JSON as property value, UI can read the content and use it as application’s config directly

Option - 2

Since the UI is the one that creates the dataprep app, there can be a setting in the static UI configuration for the dataprep config object that should be used when creating the app. The UI did something similar for Tracker through "cdap-ui-config.json", so the structure is in place already.

Option - 3

Adding a new config to specify artifact, application config and programs to start after application is deployed. The path to this config file can be specified through a cdap-site.xml property.

This requires a platform change and CDAP endpoint that can read the config to perform the following

• For the specified artifactName, find the latest artifact version if its not provided 
• Create an app with the identified artifact and the provided app-config
• Start the programs configured to be started.

class CreateAppConfig {
	// name of the artifact used to create the app
	String artifactName;
	
	// artifact scope, if null - artifacts from both system scope and user artifacts in current namespace will be retrieved
	@Nullable
	String artifactScope;

	//artifact version, if null - latest artifact version will be used
	@Nullable
	String artifactVersion;
	
	// application config
	@Nullable
	String config;

	// list of programs to start 
	List<StartProgram>List<StartProgramInfo> programs;
}

class StartProgramStartProgramInfo {
	ProgramType programType;
	String programId;
}

Example config for data-prep

{
   "artifactName":"wrangler-service",
   "artifactScope":"system",
   "config":{
      "disabledTypes":[
         "file"
      ],
      "connections":[
         {
            "name":"gcsDefault",
            "type":"gcs",
            "properties":{
               "project":"cdap"
            }
         },
         {
            "name":"bqDefault",
            "type":"bigquery",
            "properties":{

            }
         }
      ]
   },
   "programs":[
      {
         "programType":"Service",
         "programId":"wrangler-service"
      }
   ]
}

The CreateAppConfig is specified as a JSON file, The path for this file can be configured using "cdap-site.xml" property. 

<property>
<name>dataprep.config.path</name>
<value></value>
<description> Path to the configuration file for creating and configuring dataprep app</description>
</property>

Notes

• This decouples additional steps in UI for enabling data-prep from the UI, as the backend can perform finding latest artifact, app-creation and starting programs.
• Administrator can make changes to the config file and the changes will be read on re-deploy of the app without requiring a CDAP restart. 

Approach #2 Data prep code to handle connection type classification (considered)

Data-prep ConnectionType class will have the logic specifying the default modes they are supported on and the modes they are not supported upon per connection type.

public enum ConnectionType { 
 FILE("file", Collections.singletonList(Mode.NATIVE), Collections.singletonList(Mode.GCP)),
 DATABASE("database"),
 TABLE("table"),
 S3("s3", Collections.singletonList(Mode.AWS), Collections.emptyList()),
 GCS("gcs", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 BIGQUERY("bigquery", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 KAFKA("kafka");

 private String type;
 Map<Mode, List<String>> defaultTypes = new HashMap<>();
 Map<Mode, List<String>> configurableType = new HashMap<>();

 ConnectionType(String type, List<Mode> defaultModes, List<Mode> disabledModes) {
	// logic to update default and configurable type map
 }
}

Based on the above points, Approach#1 seems like a better approach for configuring connection-types and connections.

Listing connections

The listing connections endpoint returns all the connections across connection types.

GET : connections?type=<connection-type>

If we want to allow deleting default connections, no changes are required in the list connections response, else if we want to disallow we need to add a flag to indicate if the connection can be deleted or not. 

Example

[
  {
    // existing fields
    created:1533856371,
    description:"kafka connection",
    id:"kafka2",
    name:"Kafka2",
    type:"kafka",
    Updated:1533856371,

    // newly added fields
    "canDelete" : true,
    "canEdit" : true
  },
  {
    // existing fields
    created:1533856371,
    description:"GCS connection",
    id:"gcsdefault",\
    name:"gcsdefault",
    type:"gcs",
    Updated:1533856371,

    // newly added field
    // disabling delete
    "canDelete" : false,
	"canEdit" : false
  },
  ...
]