Option #3 

Story 1 - Schema and field value suggestions : 

Plugin annotation @Endpoint: 

REST API :

POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensionstypes/
{plugin-type}/
plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}&pluginScope={plugin-scope} 

 
Request-Body :  Map<String,JSON String> -  fieldName to value mapping.
 
Response : 
200, Successful Response JSON string

@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {

  /**
   * Returns the endpoint.
   */
  String endpoint();
}

@Endpoint("listTables")
List<String> listTables(ListTableRequest request)

@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request)	

Story 2 - Drafts

HTTP Request Type

Endpoint

Request Body

Response Status

Response Body

PUT

/namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/

{

"config": {...}

}

200 OK: config saved successfully

409 CONFLICT: draft-name already exists

500 Error: while saving the draft

POST

/namespaces/{namespace-id}/configurations/{config-id}/versions

{

"config ": {...}

}

200 OK: config updated successfully

404 NOT Found : config doesn't exist already, cannot be updated.

500 Error while updating the config

GET

404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)
 
Description : In the request we refer to the plugin-artifact and not the parent artifact. we could use one of the available parent artifact. 

@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {

  /**
   * Returns the endpoint.
   */
  String endpoint();
}

@Endpoint("listTables")
List<String> listTables(ListTableRequest request)

@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request)	

Story 2 - Drafts 

Configurations HTTP Handler:

Single HTTP Handler for unifying Console Setting Handler and Dashboards HTTP Handler. 

The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.

Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:

 // create/update defaults this include user's plugin version preferences, etc.
 PUT : namespaces/{namespace-id}/defaults -d '@default.json' 
 GET : namespaces/{namespace-id}/defaults 

JAVA API - Config Store:

"Drafts", "Plugin Templates",  "Default versions"  and "Dashboards" are type of configurations specified as  "config-type" in the REST call.

The individual JSON-config or object would be identified by "object-id".

JAVA API - Config Store:

void create(String namespace, String type, Config config) throws ConfigExistsException;

void deletecreateOrUpdate(String namespace, String type, StringConfig idconfig) throws ConfigNotFoundException;

List<Config> void delete(String namespace, String type, String id) throws ConfigNotFoundException;

List<Config> list(String namespace, String type);

Config get(String namespace, String type, String id) throws ConfigNotFoundException; 

void update(String namespace, String type, Config config) throws ConfigNotFoundException;

// get a particular version of an entry. 
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; 
// get all the versions of an entry.
List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; 
 

Open Questions :

1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset? 

2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?

Schema Propagation and Validation through backend - DryRuns:

POST : namespace/{namespace-id}/dry-run 

Request-Body : JSON Config.

Response-Body: 
JSON Config with additional fields in the plugin for output schema, 
exceptions in configuring pipeline stage, etc.  

User Stories (3.5.0)

1. For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
2. query for plugins available for a certain artifacts and list them in UI
3. obtaining output schema of plugins provided the input configuration information
4. deploying pipeline and start/stop the pipeline
5. query the status of a pipeline run and current status of execution if there are multiple stages.
6. get the next schedule of run, ability to query metrics and logs for the pipeline runs.
7. creating and saving pipeline drafts
8. get the input/output streams/datasets of the pipeline run and list them in UI. 
9. explore the data of streams/datasets used in the pipeline if they are explorable. 
10. Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
11. delete hydrator pipeline
12. the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.  

Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features. 

Generic Endpoints