...
Option #3
Story 1 - Schema and field value suggestions :
Plugin annotation @Endpoint:
REST API :
Code Block |
---|
POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/
{plugin-type}/plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}&pluginScope={plugin-scope}
Request-Body : Map<String, String> - fieldName to value mapping.
Response : 200, JSON string |
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Story 2 - Drafts
HTTP Request Type | Endpoint | Request Body | Response Status | Response Body | |
POST | /namespaces/{namespace-id}/configurations/{config-id}/ |
| 200 OK: config saved successfully 409 CONFLICT: draft-name already exists 500 Error: while saving the draft | ||
PUT | /namespaces/{namespace-id}/configurations/{config-id}/ |
| 200 OK: config updated successfully 404 NOT Found : config doesn't exist already, cannot be updated. 500 Error while updating the config | ||
GET | /namespaces/{namespace-id}/configurations/{config-id}/ | 200 return all the versions for the config identified by the config-name 404 config not found 500 error while getting config |
| ||
GET | /namespaces/{namespace-id}/configurations/{config-id}/versions/{version-number} -1 -> latest version | 200 return the versions for the config identified by the config-id and version-number 404 config with version found 500 error while getting config |
| ||
GET | /namespaces/{namespace-id}/configurations/ | 200 return the name of list of all saved configs 500 error | [ | ||
DELETE | /namespaces/{namespace-id}/configurations/ | 200 successfully deleted all configs 500 error while deleting | |||
DELETE | /namespaces/{namespace-id}/configurations/{config-id} | 200 successfully deleted the specified config 404 config does not exist 500 error while deleting |
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
Code Block | ||
---|---|---|
| ||
GET namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ // create a new plugin template POST namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // update existing plugin template PUT namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // delete the plugin template DELETE namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ |
Code Block | ||
---|---|---|
| ||
// create/update defaults this include user's plugin version preferences, etc. PUT : namespaces/{namespace-id}/defaults -d '@default.json' GET : namespaces/{namespace-id}/defaults |
JAVA API - Config Store:
Code Block | ||
---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException; void createOrUpdate(String namespace, String type, Config config); void delete(String namespace, String type, String id) throws ConfigNotFoundException; List<Config> list(String namespace, String type); Config get(String namespace, String type, String id) throws ConfigNotFoundException; void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry. Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; // get all the versions of an entry. Config getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; // delete all entries of specified type. void delete(String namespace, String type) |
Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
3) Dependent UI changes.
User Stories (3.5.0)
- For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
- query for plugins available for a certain artifacts and list them in UI
- obtaining output schema of plugins provided the input configuration information
- deploying pipeline and start/stop the pipeline
- query the status of a pipeline run and current status of execution if there are multiple stages.
- get the next schedule of run, ability to query metrics and logs for the pipeline runs.
- creating and saving pipeline drafts
- get the input/output streams/datasets of the pipeline run and list them in UI.
- explore the data of streams/datasets used in the pipeline if they are explorable.
- Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
- delete hydrator pipeline
- the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.
Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.
Generic Endpoints
...