...
Option #3
Story 1 - Schema and field value suggestions :
Plugin annotation @Endpoint:
REST API :
| Code Block |
|---|
POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/
plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}
Request-Body : JSON - fieldName to value mapping.
Response :
200, Successful Response JSON string
404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)
Description : In the request we refer to the plugin-artifact and not the parent artifact. we could use one of the available parent artifact. |
| Code Block |
|---|
@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {
/**
* Returns the endpoint.
*/
String endpoint();
} |
| Code Block | ||
|---|---|---|
| ||
@Endpoint("listTables")
List<String> listTables(ListTableRequest request)
@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request) |
Story 2 - Drafts
Configurations HTTP Handler:
Single HTTP Handler for unifying Console Setting Handler and Dashboards HTTP Handler.
HTTP Request Type | Endpoint | Request Body | Response Status | Response Body | |
PUT | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/ |
| 200 OK: config object saved successfully 409 CONFLICT: config with object-id already exists 500 Error: while saving the draft | ||
POST | /namespaces/{namespace-id}/configurations{config-type}/objects/{object-id}/versions |
| 200 OK: config object updated successfully 404 NOT Found : config object doesn't exist already, cannot be updated. 500 Error while updating the config | ||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}versions | 200 return all the versions for the config identified by the object-id 404 config object not found 500 error while getting config object |
| ||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions/{version-number}
| 200 return the versions for the config object identified by the object-id and version-number 404 config object with version found 500 error while getting config object |
| ||
| GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} Get latest version | 200 return the latest version for the config object 404 config object with version found 500 error while getting the latest config object | { "timestamp" : "...", "config": { .... } }
| ||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects | 200 return the name of list of all saved config objects 500 error | [ "name" : "StreamToTPFS", "lastSaved": "..", .. } , | ||
DELETE | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} | 200 successfully deleted the specified object 404 object does not exist 500 error while deleting |
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
"Drafts", "Plugin Templates", "Default versions" and "Dashboards" are type of configurations specified as "config-type" and in the REST call.
The individual jsonJSON-config or object would be identified by "object-id".
JAVA API - Config Store:
| Code Block | ||
|---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException; void createOrUpdate(String namespace, String type, Config config); void delete(String namespace, String type, String id) throws ConfigNotFoundException; List<Config> list(String namespace, String type); Config get(String namespace, String type, String id) throws ConfigNotFoundException; void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
| Code Block | ||
|---|---|---|
| ||
// get a particular version of an entry. Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; // get all the versions of an entry. List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; |
Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
3) Dependent UI changes.
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
| Code Block |
|---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
User Stories (3.5.0)
- For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
- query for plugins available for a certain artifacts and list them in UI
- obtaining output schema of plugins provided the input configuration information
- deploying pipeline and start/stop the pipeline
- query the status of a pipeline run and current status of execution if there are multiple stages.
- get the next schedule of run, ability to query metrics and logs for the pipeline runs.
- creating and saving pipeline drafts
- get the input/output streams/datasets of the pipeline run and list them in UI.
- explore the data of streams/datasets used in the pipeline if they are explorable.
- Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
- delete hydrator pipeline
- the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.
Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.
Generic Endpoints
...