Approach #1 Data-prep app config to specify connection type classification (Preferred)

Administrators can specify a data-prep config JSON specific to their environment.

The config JSON can be used to specify a list of ConnectionTypeConfig 

• ConnectionTypeConfig can be used to specify that a connection needs to be created by default for the connection type and the properties for that connection.

• ConnectionTypeConfig can be used to specify that a connection type needs to be hidden in DataPrep

{
  "types": [ "gcs", "bigquery", "kafka", "database", "s3" ],
  "connections": [
    {
      "name": "gcsDefault",
      "type": "gcs",
      "properties": {
        "serviceFilePath": "~/gcp/service-config.json",
        "project": "cdap"
      }
    }, 
    {
      "name": "bqDefault",
      "type": "bigquery",
      "properties": { }
    },
    ...
  ]
}

Any connections specified in the config are created on start up of the Wrangler service. They cannot be deleted.

Example explanation :

• The 'file' connector type is not in the list, so no file connections are allowed.
• There is a default GCS connection that uses pre-defined credentials and project information as specified in the properties map. 
• There is a default BigQuery connection without any properties. The BigQuery connection will read required credentials from the environment.
• Kafka, Database and S3 connection types are configured to be visible, but no default connections are configured for them.

Surfacing the data-prep-config JSON to data-prep application

Data-prep config JSON can be specified through cdap-site.xml property

Option - 1

<property>
<name>data.prep.config.json</name>
<value></value>
<description>
   data prep config JSON content, this config is used to specify the connection-types that are   
   available as default and the connection-types that have to be disabled in data-prep.
</description>
</property>

Example 

<property>
<name>data.prep.config.json</name>
<value>
{
  "types": [ "gcs", "bigquery", "kafka", "database", "s3" ],
  "connections": [
    {
      "name": "gcsDefault",
      "type": "gcs",
      "properties": {
        "serviceFilePath": "~/gcp/service-config.json",
        "project": "cdap"
      }
    }, 
    {
      "name": "bqDefault",
      "type": "bigquery",
      "properties": { }
    },
    ...
  ]
}
</value>
</property>

Notes

• We have to provide JSON content instead of JSON file path, as the CDAP UI service could be running on different instance than CDAP backend service

• Currently data-prep application is deployed and run separately on each namespace, however in the future when data-prep is moved to system namespace, it will be easier to have the data-prep application configured and deployed using bootstrap config files during CDAP startup instead of specifying through a cdap-site.xml property.

• By providing the JSON as property value, UI can read the content and use it as application’s config directly

Option - 2

Since the UI is the one that creates the dataprep app, there can be a setting in the static UI configuration for the dataprep config object that should be used when creating the app. The UI did something similar for Tracker through "cdap-ui-config.json", so the structure is in place already.

Approach #2 Data prep code to handle connection type classification (considered)

Data-prep ConnectionType class will have the logic specifying the default modes they are supported on and the modes they are not supported upon per connection type.

public enum ConnectionType { 
 FILE("file", Collections.singletonList(Mode.NATIVE), Collections.singletonList(Mode.GCP)),
 DATABASE("database"),
 TABLE("table"),
 S3("s3", Collections.singletonList(Mode.AWS), Collections.emptyList()),
 GCS("gcs", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 BIGQUERY("bigquery", Collections.singletonList(Mode.GCP), Collections.emptyList()),
 KAFKA("kafka");

 private String type;
 Map<Mode, List<String>> defaultTypes = new HashMap<>();
 Map<Mode, List<String>> configurableType = new HashMap<>();

 ConnectionType(String type, List<Mode> defaultModes, List<Mode> disabledModes) {
	// logic to update default and configurable type map
 }
}

Based on the above points, Approach#1 seems like a better approach for configuring connection-types and connections.

Listing connections

The listing connections endpoint returns all the connections across connection types.

GET : connections?type=<connection-type>

If we want to allow deleting default connections, no changes are required in the list connections response, else if we want to disallow we need to add a flag to indicate if the connection can be deleted or not. 

Example

[
  {
    // existing fields
    created:1533856371,
    description:"kafka connection",
    id:"kafka2",
    name:"Kafka2",
    type:"kafka",
    Updated:1533856371,

    // newly added fields
    "canDelete" : true,
    "canEdit" : true
  },
  {
    // existing fields
    created:1533856371,
    description:"GCS connection",
    id:"gcsdefault",\
    name:"gcsdefault",
    type:"gcs",
    Updated:1533856371,

    // newly added field
    // disabling delete
    "canDelete" : false,
	"canEdit" : false
  },
  ...
]