...
Option #3
Story 1 - Schema and field value suggestions :
Plugin annotation @Endpoint:
REST API :
Code Block |
---|
POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/ {plugin-type}/plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}&pluginScope={plugin-scope} Request-Body : Map<String, String> - fieldName to value mapping. Response : 200, JSON string |
Code Block | ||
---|---|---|
| ||
@Endpoint("listTable") List<String> listTables(Map<String, String> inputConfigString jdbcPluginType, String jdbcPluginName, String connectionString, @Nullable String user, @Nullable String password) @Endpoint("getSchema") Map<String, String> getSchema(Map<String, String> inputConfigString jdbcPluginType, String jdbcPluginName, String connectionString, String importQuery, @Nullable String user, @Nullable String password) |
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Code Block |
---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
Story 2 - Drafts
HTTP Request Type
Endpoint
Request Body
Response Status
Response Body
Story 2 - Drafts
HTTP Request Type | Endpoint | Request Body | Response Status | Response Body | ||
POST | /namespaces/{namespace-id}/configurations/{config-id}/ |
| 200 OK: config saved successfully 409 CONFLICT: draft-name already exists 500 Error: while saving the draft | |||
PUT | /namespaces/{namespace-id}/configurations/{config-id}/ |
| 200 OK: config updated successfully 404 NOT Found : config doesn't exist already, cannot be updated. 500 Error while updating the config | |||
GET | /namespaces/{namespace-id}/configurations/{config-id}/ | 200 return all the versions for the config identified by the config-name 404 config not found 500 error while getting config |
| |||
GET | /namespaces/{namespace-id}/configurations/{config-id}/versions/{version-number}
| |||||
{ "config": {...} } | 200 OK: config saved successfully 409 CONFLICT: draft-name already exists 500 Error: while saving the draft | PUT | /namespaces/{namespace-id}/configurations/{config-id}/ | |||
{ "config ": {...} } | 200 OK: config updated successfully 404 NOT Found : config doesn't exist already, cannot be updated. 500 Error while updating the config | -1 -> latest version | 200 return the versions for the config identified by the config-id and version-number 404 config with version found 500 error while getting config |
| ||
GET | /namespaces/{namespace-id}/configurations/{config-id}/ | 200 return all the versions for the config identified by the config-name404 config not foundname of list of all saved configs 500 error while getting config | [ "streamToTPFS", DBToHBase", "config": { "source" : { .... }, "transforms" : [...], "sinks" [...] "connections" : [..] } }, ... ] | GET | / | |
DELETE | /namespaces/{namespace-id}/configurations/ | 200 successfully deleted all configs 500 error while deleting | ||||
DELETE | /namespaces/{namespace-id}/configurations/{config-id}/versions/{version-number}-1 -> latest version | 200 return the versions for the config identified by the config-id and version-numbersuccessfully deleted the specified config 404 config with version founddoes not exist 500 error while getting configdeleting | ||||
{ "timestamp" : "...", "config": { "source" : { .... }, "transforms" : [...], "sinks" [...] "connections" : [..] } } | GET | /
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
Code Block | ||
---|---|---|
| ||
GET namespaces/{namespace-id}/ |
200 return the name of list of all saved configs
500 error
[
"streamToTPFS",
"DBToHBase",
...
]
DELETE
plugin-templates/{plugin-template-id}/ // create a new plugin template POST namespaces/{namespace-id}/ |
200 successfully deleted all configs
500 error while deleting
DELETE
plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // update existing plugin template PUT namespaces/{namespace-id}/ |
plugin-templates/{ |
plugin-template-id} |
200 successfully deleted the specified config
404 config does not exist
500 error while deleting
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
Code Block | ||
---|---|---|
| ||
GET/ -d '@plugin-template.json' // delete the plugin template DELETE namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ |
Code Block | ||
---|---|---|
| ||
// create/update defaults this include user's plugin version preferences, etc. PUT : namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ // create a new plugin template POST defaults -d '@default.json' GET : namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // update existing plugin template PUT namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // delete the plugin template DELETE namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ |
Code Block | ||
---|---|---|
| ||
// create/update defaults this include user's plugin version preferences, etc.
PUT : namespaces/{namespace-id}/defaults -d '@default.json'
GET : namespaces/{namespace-id}/defaults |
JAVA API - Config Store:
Code Block | ||
---|---|---|
| ||
void createdefaults |
JAVA API - Config Store:
Code Block | ||
---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException; void createOrUpdate(String namespace, String type, Config config); void delete(String namespace, String type, String id) throws ConfigNotFoundException; List<Config> list(String namespace, String type); Config get(String namespace, String type, String id) throws ConfigNotFoundException; void update(String namespace, String type, Config config) throws ConfigExistsException; void createOrUpdate(String namespace, String type, Config config); void deleteConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry. Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; // get List<Config> listall the versions of an entry. Config getAllVersions(String namespace, String type); Config get(String namespace, String type, String id) throws ConfigNotFoundException; // delete all entries of specified type. void updatedelete(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry.
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException;
// get all the versions of an entry.
Config getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException;
// delete all entries of specified type.
void delete(String namespace, String type) |
Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
3) Dependent UI changes.Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
3) Dependent UI changes.
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Code Block |
---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
User Stories (3.5.0)
- For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
- query for plugins available for a certain artifacts and list them in UI
- obtaining output schema of plugins provided the input configuration information
- deploying pipeline and start/stop the pipeline
- query the status of a pipeline run and current status of execution if there are multiple stages.
- get the next schedule of run, ability to query metrics and logs for the pipeline runs.
- creating and saving pipeline drafts
- get the input/output streams/datasets of the pipeline run and list them in UI.
- explore the data of streams/datasets used in the pipeline if they are explorable.
- Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
- delete hydrator pipeline
- the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.
Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.
Generic Endpoints
...