Endpoint

Response Status

/extensions/hydrator/drafts/{draft-name}

{ 
"config": {...},
"message" : "..."
}

200 OK: draft created and saved successfully

409 CONFLICT: draft-name already exists

500 Error: while creating the draft

{ 
"config": {...},
"message" : "..."
}

200 OK: draft updated successfully

404 NOT Found : draft doesn't exist already, cannot be updated.

400 BAD Request : only 20 versions can be stored, delete old version before storing.

500 Error while updating the draft

/extensions/hydrator/drafts/{draft-name}/versions/

200 return all the versions for the draft identified by the draft-name

404 draft not found

500 error while getting draft

[
{
"message" : "...",
"config": {
 "source" : {
    ....
  }, 
 "transforms" : [...],
 "sinks" [...]
 "connections" : [..]
 }
}, 
... 
]

/extensions/hydrator/drafts/{draft-name}/versions/{version-number}

-1 -> latest version

200 return the versions for the draft identified by the draft-name and version-number

404 draft not found

500 error while getting draft

{ 
"message" : "...",
"config": {
 "source" : {
    ....
  }, 
 "transforms" : [...],
 "sinks" [...]
 "connections" : [..]
 }
}

200 return the list of all saved drafts

500 error

[ 
  "streamToTPFS",
  "DBToHBase",
   ...
]

200 successfully deleted all drafts

500 error while deleting

200 successfully deleted the specified draft

404 draft does not exist

500 error while deleting

200 successfully deleted the version of a drat

404 draft with the version does not exist

500 error while deleting

/extensions/hydrator/plugins/{plugin-name}/schema

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
},
"jdbcConnectionString": "...", 
 "jdbcPluginName": "...", 
 "tableName" : "..."
}

200 based on the plugin and plugin-properties

 determine output schema and return output schema

404 unrecognized plugin-name

500 Error

{ 
  "field1" : Integer,
  "field2" : String,
   ...
  "fieldN" : Double
}

/extensions/hydrator/plugins/{plugin-name}/list

QueryParam : target

Example: target=table

Example:

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
}
"connectionString":   ..., 
 "username": ..., 
 "password" : ...
}

For the specified plugin, based on the provided connection information, get the list of available target field and return the list.

200 , list of available values for target type field. Example: list of tables in a database.

500 error while retrieving.

[ 
  "tableA",
  "tableB"
   ...
  "tableN" 
]

Design

Option #1

Description

The hydrator app needs to be able to write/read to a dataset to store and retrieve drafts and other information about business logic.  We can implement a Hydrator CDAP Application with a service that can have REST endpoints to serve the required hydrator functionalities. Enabling Hydrator in a namespace will deploy this Hydrator app and start the service. Hydrator UI would ping for this service to be available before coming up. The back-end business logic actions which directly needs to use the CDAP services endpoints can be made generic. 

• Pros

  • Everything (Drafts, etc) stored in the same namespace, proper cleanup when namespace is deleted.

• Cons

  • Every namespace will have an extra app for supporting hydrator if hydrator is enabled. Running this service, will run 2 containers per namespace. we can add an option to enable/disable hydrator if we are not using hydrator in a namespace.  It might feel weird as a user app, as the user didn't write/create this app.  

Option #2

Description

We will still use an Hydrator CDAP app but we create an "Extensions" namespace and have the "hydrator" app only deployed in the "extensions" namespace, this app would serve the hydrator requests for all namespaces.

• Pros

  • Less amount of resources used, only 2 container's used rather than 2 container’s per namespace, only one dataset is used.
  • Only one app for using hydrator across namespace and not an app per namespace, less clutter. 
  • New extensions could be added to the same namespace to support other use cases in future.

• Cons

  • Using a single dataset for storing all drafts across namespace is less secure?.
  • User won't be able to create a new namespace called "Extensions", as it will be reserved.

Open Questions

• How to delete the drafts when the namespace is deleted ?
• When to stop this service? 
• Availability of the service? 
• Security
  • If we decide to add more capability in hydrator back-end app, Eg: Make the pipeline validation/deploy app, etc,  then in secure environment, 
  • The hydrator-service can discover appropriate cdap.service and call appropriate endpoints?

Option #3 (based on discussion with terence)

 No new user level apps are deployed. Preference store is used to store user drafts of hydrator apps.

'configurePipeline' can be changed to return partial results, it can return pluginSpecification with possible values for missing information in plugin config, the pluginSpecification will be serialized into applicationSpecification and returned to the user.  

Example:

1. Hydrator makes a call to Preference store to save name-spaced draft, in order to delete the drafts, delete endpoint is called in preference store for the drafts. If user deletes the namespace manually from CDAP-CLI, the preference store drops everything in that namespace including the drafts.

2. Plugin configure stage will accept incomplete config and will create PluginSpecification, with possible values for incomplete config.

1. Example : User is using a DBSource plugin, he provides connectionString, userName and password. the UI hits /validate endpoint with config, DBSource’s configurePlugin is called, it inspects the config, notices the required field ‘tableName' is missing, it connects to the database and gets the list of table names, writes this list in PluginSpecification and returns failure.

2. User notices the failure, reads the specification to get the list of tables, selects the table he is interested in and makes the same call again, DBSource’s configure plugin notices schema is missing and ‘import’ field is missing. It then populates schema information in spec and returns failure.

3. user fills the ‘import’, ‘count’ queries and changes schema appropriately and makes the same call, all the necessary fields are present and valid, the DBSource plugin returns successful for this stage. user proceeds to next stage.

REST API:

HTTP Request Type

Endpoint

Request Body

Response Status

Response Body

POST

/namespaces/{namespace-id}/drafts/{draft-id}/

{

"config": {...}

}

200 OK: draft created and saved successfully

409 CONFLICT: draft-name already exists

500 Error: while creating the draft

PUT

/namespaces/{namespace-id}/drafts/{draft-id}/

{

"config ": {...}

}

200 OK: draft updated successfully

404 NOT Found : draft doesn't exist already, cannot be updated.

500 Error while updating the draft

GET

/namespaces/{namespace-id}/drafts/{draft-id}/

200 return all the versions for the draft identified by the draft-name

404 draft not found

500 error while getting draft

[

{

"timestamp" : "...",

"config": {

"source" : {

   ....

 },

"transforms" : [...],

"sinks" [...]

"connections" : [..]

}

},

...

]

GET

/namespaces/{namespace-id}/drafts/{draft-id}/versions/{version-number}

-1 -> latest version

200 return the versions for the draft identified by the draft-name and version-number

404 draft with version found

500 error while getting draft

{

"timestamp" : "...",

"config": {

"source" : {

   ....

 },

"transforms" : [...],

"sinks" [...]

"connections" : [..]

}

}

GET

/namespaces/{namespace-id}/drafts/

200 return the name of list of all saved drafts

500 error

[
 "streamToTPFS",
 "DBToHBase",
  ...
]

DELETE

/namespaces/{namespace-id}/drafts/

200 successfully deleted all drafts

500 error while deleting

DELETE

/namespaces/{namespace-id}/drafts/{draft-id}

200 successfully deleted the specified draft

404 draft does not exist

500 error while deleting

The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.

ConsoleSettingsHttpHandler->ConfigStore.

Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:

1) pre-configured plugin templates. 

Endpoints :  

GET namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ 

POST namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' -> create a new plugin template

PUT namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' -> update existing plugin template

2) default versions of plugins (user preferences)

Endpoints: 

 PUT : namespaces/{namespace-id}/defaults -d '@default.json' -> create/update defaults this include user's plugin version preferences, etc.

 GET : namespaces/{namespace-id}/defaults -> Gets defaults configured by user.

ConfigStore Existing methods  :

void create(String namespace, String type, Config config) throws ConfigExistsException;

void createOrUpdate(String namespace, String type, Config config);

void delete(String namespace, String type, String id) throws ConfigNotFoundException;

List<Config> list(String namespace, String type);

Config get(String namespace, String type, String id) throws ConfigNotFoundException; 

void update(String namespace, String type, Config config) throws ConfigNotFoundException;

ConfigStore new methods:

Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; // get a version of a draft
Config getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; // get all the versions of the draft. 
void delete(String namespace, String type) // type-> drafts, delete all drafts in the namespace.

Existing Config class: 

public final class Config {
 // draft-id
 private final String id;  
 // config -> json-config and other properties, example:timestamp -> currentTime.
 private final Map<String, String> properties; 


}

Questions :

1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, would this affect the "preferences" stored by PreferenceStore?. This would also need CDAP-upgrade to update properties for the existing dataset? 

REST API for configure suggestions - AppFabric :

Request-Method : POST

Request-Endpoint : /namespaces/{namespace-id}/apps/{app-id}/configure

Request-Body : config-JSON 

{
    "artifact": {
        "name": "cdap-etl-batch",
        "scope": "SYSTEM",
        "version": "3.4.0-SNAPSHOT"
    },
    "name": "pipeline",
    "config": {
        "source": {
			     "name": "Stream",
                 "plugin": {
                    "name": "StreamSource",
                    "artifact": {
                        "name": "core-plugins",
                        "version": "1.3.0-SNAPSHOT",
                        "scope": "SYSTEM"
                    },
                    "properties": {
                        "format": "syslog",
                        "name": "test",
                        "duration": "1d"
                    }
                }
            },
         "sinks" : [{..}],
          "transform": [{..}, {...}]
        }
}

Response-Body : Config JSON

{
    "artifact": {
        "name": "cdap-etl-batch",
        "scope": "SYSTEM",
        "version": "3.4.0-SNAPSHOT"
    },
    "name": "pipeline",
    "config": {
        "source": {
				"name": "Stream",
                "plugin": {
                    "name": "StreamSource",
                    "artifact": {
                        "name": "core-plugins",
                        "version": "1.3.0-SNAPSHOT",
                        "scope": "SYSTEM"
                    },
                    "properties": {
                        "format": "syslog",
                        "name": "test",
                        "duration": "1d",
                        "suggestions" : [{ 
                             "schema" : [ 
                                 { 
								 	"ts" : "long", 
                                    "headers", "Map<String, String>", 
                                    "program", "string",
									"message":"string",
									"pid":"string"
						         }
						       ]
							}],
						"isComplete" : "false"
                  	}
                }
            },
         "sinks" : [{..}],
          "transform": [{..}, {...}]
        }
}

Plugin API Change

@Beta
public interface PipelineConfigurable {
  // change in return-type.
  ConfigResponse configurePipeline(PipelineConfigurer pipelineConfigurer) throws IllegalArgumentException; 
}

public class ConfigResponse extends Config {
 // list of suggestions for fields. 
 List<Suggestion> suggestions;
 // if there were any exception while executing configure 
 @Nullable
 String exception;
 // is the stage configuration complete ? 
 @DefaultValue("false")
 boolean isComplete;
}

public class Suggestion {
String fieldName;
// list of possible values for the fieldName
List<String> fieldValues; 
} 

@Beta
public interface ApplicationContext<T extends Config> {
  // existing
  T getConfig();
  // application will set a config response
  void setResponseConfig(T response);
  // get the response config
  T getResponseConfig();
}

Questions 

Design

Option #1

Description

The hydrator app needs to be able to write/read to a dataset to store and retrieve drafts and other information about business logic.  We can implement a Hydrator CDAP Application with a service that can have REST endpoints to serve the required hydrator functionalities. Enabling Hydrator in a namespace will deploy this Hydrator app and start the service. Hydrator UI would ping for this service to be available before coming up. The back-end business logic actions which directly needs to use the CDAP services endpoints can be made generic. 

• Pros

  • Everything (Drafts, etc) stored in the same namespace, proper cleanup when namespace is deleted.

• Cons

  • Every namespace will have an extra app for supporting hydrator if hydrator is enabled. Running this service, will run 2 containers per namespace. we can add an option to enable/disable hydrator if we are not using hydrator in a namespace.  It might feel weird as a user app, as the user didn't write/create this app.  

Option #2

Description

We will still use an Hydrator CDAP app but we create an "Extensions" namespace and have the "hydrator" app only deployed in the "extensions" namespace, this app would serve the hydrator requests for all namespaces.

• Pros

  • Less amount of resources used, only 2 container's used rather than 2 container’s per namespace, only one dataset is used.
  • Only one app for using hydrator across namespace and not an app per namespace, less clutter. 
  • New extensions could be added to the same namespace to support other use cases in future.

• Cons

  • Using a single dataset for storing all drafts across namespace is less secure?.
  • User won't be able to create a new namespace called "Extensions", as it will be reserved.

Open Questions

• How to delete the drafts when the namespace is deleted ?
• When to stop this service? 
• Availability of the service? 
• Security
  • If we decide to add more capability in hydrator back-end app, Eg: Make the pipeline validation/deploy app, etc,  then in secure environment, 
  • The hydrator-service can discover appropriate cdap.service and call appropriate endpoints?

Option #3 

Story 1 - Schema and field value suggestions : 

Plugin annotation @Endpoint: 

REST API :

POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/types/{plugin-type}
plugins/{plugin-name}/methods/{plugin-method}?scope={artifact-scope} 

 
Request-Body :  JSON -  fieldName to value mapping.
 
Response : 
200, Successful Response JSON string
404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)
 
Description : In the request we refer to the plugin-artifact and not the parent artifact. we could use one of the available parent artifact. 

@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {

  /**
   * Returns the endpoint.
   */
  String endpoint();
}

@Endpoint("listTables")
List<String> listTables(ListTableRequest request)

@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request)	

Story 2 - Drafts 

Configurations HTTP Handler:

Single HTTP Handler for unifying Console Setting Handler and Dashboards HTTP Handler. 

"Drafts", "Plugin Templates",  "Default versions"  and "Dashboards" are type of configurations specified as  "config-type" in the REST call.

The individual JSON-config or object would be identified by "object-id".

JAVA API - Config Store:

void create(String namespace, String type, Config config) throws ConfigExistsException;

void createOrUpdate(String namespace, String type, Config config);

void delete(String namespace, String type, String id) throws ConfigNotFoundException;

List<Config> list(String namespace, String type);

Config get(String namespace, String type, String id) throws ConfigNotFoundException; 

void update(String namespace, String type, Config config) throws ConfigNotFoundException;

// get a particular version of an entry. 
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; 
// get all the versions of an entry.
List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; 
 

Schema Propagation and Validation through backend - DryRuns:

POST : namespace/{namespace-id}/dry-run 

Request-Body : JSON Config.

Response-Body: 
JSON Config with additional fields in the plugin for output schema, 
exceptions in configuring pipeline stage, etc.  

User Stories (3.5.0)

1. For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
2. query for plugins available for a certain artifacts and list them in UI
3. obtaining output schema of plugins provided the input configuration information
4. deploying pipeline and start/stop the pipeline
5. query the status of a pipeline run and current status of execution if there are multiple stages.
6. get the next schedule of run, ability to query metrics and logs for the pipeline runs.
7. creating and saving pipeline drafts
8. get the input/output streams/datasets of the pipeline run and list them in UI. 
9. explore the data of streams/datasets used in the pipeline if they are explorable. 
10. Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
11. delete hydrator pipeline
12. the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.  

Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features. 

Generic Endpoints