Approach #1 Data-prep app config to specify supported connection types and default connections (Preferred)

Administrators can specify a data-prep config JSON specific to their environment.

The config JSON can be used to specify a list of ConnectionTypeConfig 

• ConnectionTypeConfig can be used to specify that a connection needs to be created by default for the connection type and the properties for that connection.

• ConnectionTypeConfig can be used to specify the  set of connection types that are supported in DataPrep

{
   "disabledTypes":[
      "file"
   ],
   "connections":[
      {
         "name":"gcsDefault",
         "type":"gcs",
         "properties":{
            "nameserviceFilePath": "gcsDefault"~/gcp/service-config.json",
      "type      "project": "gcscdap",
      "properties": {  }
      "serviceFilePath": "~/gcp/service-config.json"},
        "project": "cdap"
      }
    }, 
{
   {       "name": "bqDefault",
         "type": "bigquery",
         "properties":{
{
    }     },
     ... }
 ]
}  ], 
   "defaultConnection":{
      "name":"gcsDefault",
      "type":"gcs"
   }
}

Any connections specified in the config are created on start up of the Wrangler service. They cannot be deleted.

Example explanation :

• The 'file' connector type is in disabled connection type list, so no file connections are allowed.
• There is a default GCS connection that uses pre-defined credentials and project information as specified in the properties map. 
• There is a default BigQuery connection without any properties. The BigQuery connection will read required credentials from the environment.
• Kafka, Database and S3 connection types are configured to be visible, but no default connections are configured for them.

• GCS is configured as default connection - used by dataprep UI to show this connection as the default page on data-prep

Surfacing the data-prep-config JSON to data-prep application

Data-prep config JSON can be specified through cdap-site.xml property

Option - 1

<property>
<name>data.prep.config.json</name>
<value></value>
<description>
   data prep config JSON content, this config is used to specify the connection-types that are   
   available as default and the connection-types that have to be disabled in data-prep.
</description>
</property>

Example 

<property>
<name>data.prep.config.json</name>
<value>
{
  "disabledTypes": [ "file" ],
  "connections": [
    {
      "name": "gcsDefault",
      "type": "gcs",
      "properties": {
        "serviceFilePath": "~/gcp/service-config.json",
        "project": "cdap"
      }
    }, 
    {
      "name": "bqDefault",
      "type": "bigquery",
      "properties": { }
    },
    ...
  ]
}
</value>
</property>

Notes

• We have to provide JSON content instead of JSON file path, as the CDAP UI service could be running on different instance than CDAP backend service

• Currently data-prep application is deployed and run separately on each namespace, however in the future when data-prep is moved to system namespace, it will be easier to have the data-prep application configured and deployed using bootstrap config files during CDAP startup instead of specifying through a cdap-site.xml property.

• By providing the JSON as property value, UI can read the content and use it as application’s config directly

Option - 2

Since the UI is the one that creates the dataprep app, there can be a setting in the static UI configuration for the dataprep config object that should be used when creating the app. The UI did something similar for Tracker through "cdap-ui-config.json", so the structure is in place already.

Option - 3

Adding a new CDAP endpoint that can be used to create application based on a specified artifact and start programs in the application.  

config to specify artifact, application config and programs to start after application is deployed. The path to this config file can be specified through a cdap-site.xml property.

This requires a platform change and CDAP endpoint that can read the config to perform the following

• For the specified artifactName, find the latest artifact version if its not provided 
• Create an app with the identified artifact and the provided app-config
• Start the programs configured to be started.

class CreateAppConfig {
	// name of the artifact used to create the app
	String artifactName;
	
	// artifact scope, if null - artifacts from both system scope and user artifacts in current namespace will be retrieved
	@Nullable
	String artifactScope;

	//artifact version, if null - latest artifact version will be used
	@Nullable
	String artifactVersion;
	
	// application config
	@Nullable
	String config;


	// list of programs to start 
	List<StartProgram>List<StartProgramInfo> programs;
}


class StartProgramStartProgramInfo {
	ProgramType programType;
	String programId;
}

Example config for data-prep

{
   "artifactName":"wrangler-service",
   "artifactScope":"system",
   "config":{
      "disabledTypes":[
         "file"
      ],
      "connections":[
         {
            "name":"gcsDefault",
            "type":"gcs",
            "properties":{
               "project":"cdap"
            }
         },
         {
            "name":"bqDefault",
            "type":"bigquery",
            "properties":{

            }
         }
      ]
   },
   "programs":[
      {
         "programType":"Service",
         "programId":"wrangler-service"
      }
   ]
}

The CreateAppConfig can be specified in a file, The path for this file can be configured using "cdap-site.xml" property. 

<property>
<name>dataprep.config.path</name>
<value></value>
<description> Path to the configuration file for creating and configuring dataprep app</description>
</property>

Notes

• This decouples additional steps in UI for enabling data-prep from the UI, as the backend can perform finding latest artifact, app-creation and starting programs.
• Administrator can make changes to the config file and the changes will be read on re-deploy of the app without requiring a CDAP restart. 

Approach #2 Data prep code to handle connection type classification (considered)

Data-prep ConnectionType class will have the logic specifying the default modes they are supported on and the modes they are not supported upon per connection type.

public enum ConnectionType { 
 FILE("file", Collections.singletonList(Mode.NATIVE), Collections.singletonList(Mode.GCP)),
 DATABASE("database"),
 TABLE("table"),
 S3("s3", Collections.singletonList(Mode.AWS), Collections.emptyList()),
 GCS("gcs", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 BIGQUERY("bigquery", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 KAFKA("kafka");

 private String type;
 Map<Mode, List<String>> defaultTypes = new HashMap<>();
 Map<Mode, List<String>> configurableType = new HashMap<>();

 ConnectionType(String type, List<Mode> defaultModes, List<Mode> disabledModes) {
	// logic to update default and configurable type map
 }
}

Comparison of approaches 

Approach-1

Approach-2

Flexibility

By configuring data-prep through config json specific to an environment, its very flexible to be used/configured across environments

Not flexible and would require code changes to support changes in default connection types

Extensibility

Its extensible to add new capabilities, offers administrators a way to disable connection types they they like to hide in data-prep in their cluster or environment.

Not extensible

Ease of use

Needs appropriate changes in cdap-site from administrators to configure data-prep connection-types for a certain environment.  

No config changes required from admin.

Based on the above points, Approach#1 seems like a better approach for configuring connection-types and connections.

Listing connections

The listing connections endpoint returns all the connections across connection types.

GET : connections?type=<connection-type>

If we want to allow deleting default connections, no changes are required in the list connections response, else if we want to disallow we need to add a flag to indicate if the connection can be deleted or not. 

Example

[
  {
    // existing fields
    created:1533856371,
    description:"kafka connection",
    id:"kafka2",
    name:"Kafka2",
    type:"kafka",
    Updated:1533856371,

    // newly added fields
    "canDelete" : true,
    "canEdit" : true
  },
  {
    // existing fields
    created:1533856371,
    description:"GCS connection",
    id:"gcsdefault",\
    name:"gcsdefault",
    type:"gcs",
    Updated:1533856371,

    // newly added field
    // disabling delete
    "canDelete" : false,
	"canEdit" : false
  },
  ...
]         "type":"gcs",
            "properties":{
               "project":"cdap"
            }
         },
         {
            "name":"bqDefault",
            "type":"bigquery",
            "properties":{

            }
         }
      ]
   },
   "programs":[
      {
         "programType":"Service",
         "programId":"wrangler-service"
      }
   ]
}

The CreateAppConfig is specified as a JSON file, The path for this file can be configured using "cdap-site.xml" property. 

<property>
<name>dataprep.config.path</name>
<value></value>
<description> Path to the configuration file for creating and configuring dataprep app</description>
</property>

Notes

• This decouples additional steps in UI for enabling data-prep from the UI, as the backend can perform finding latest artifact, app-creation and starting programs.
• Administrator can make changes to the config file and the changes will be read on re-deploy of the app without requiring a CDAP restart. 

Approach #2 Data prep code to handle connection type classification (considered)

Data-prep ConnectionType class will have the logic specifying the default modes they are supported on and the modes they are not supported upon per connection type.

public enum ConnectionType { 
 FILE("file", Collections.singletonList(Mode.NATIVE), Collections.singletonList(Mode.GCP)),
 DATABASE("database"),
 TABLE("table"),
 S3("s3", Collections.singletonList(Mode.AWS), Collections.emptyList()),
 GCS("gcs", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 BIGQUERY("bigquery", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 KAFKA("kafka");

 private String type;
 Map<Mode, List<String>> defaultTypes = new HashMap<>();
 Map<Mode, List<String>> configurableType = new HashMap<>();

 ConnectionType(String type, List<Mode> defaultModes, List<Mode> disabledModes) {
	// logic to update default and configurable type map
 }
}

Based on the above points, Approach#1 seems like a better approach for configuring connection-types and connections.

Modifying Connections - Administrators and Users

1. Users will be allowed to edit or delete default connections similar to other connections.
2. Administrators can change the configuration of default connections, on re-deploy, the default connections which doesn't exist already will be created.
3. Changes to existing default connections or removal of a default connection by administrator won't affect already existing connections for users in data-prep, as it can cause disruption for users who are actively using those connections.