...
Option #3
Story 1 - Schema and field value suggestions :
Jira Legacy server Cask Community Issue Tracker serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-5149
Plugin annotation @Endpoint:
REST API :
Code Block |
---|
POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensionstypes/ {plugin-type}/ plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}&pluginScope={plugin-scope} Request-Body : Map<String,JSON String> - fieldName to value mapping. Response : 200, Successful Response JSON string |
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Code Block |
---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
Story 2 - Drafts
HTTP Request Type
Endpoint
404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)
Description : In the request we refer to the plugin-artifact and not the parent artifact. we could use one of the available parent artifact. |
Code Block |
---|
@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {
/**
* Returns the endpoint.
*/
String endpoint();
} |
Code Block | ||
---|---|---|
| ||
@Endpoint("listTables")
List<String> listTables(ListTableRequest request)
@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request) |
Story 2 - Drafts
Jira Legacy server Cask Community Issue Tracker serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-5154
Configurations HTTP Handler:
Single HTTP Handler for unifying Console Setting Handler and Dashboards HTTP Handler.
HTTP Request Type | Endpoint : (Table Assumes we are using config-type -> drafts) | Request Body | Response Status | Response Body | |||||
POSTPUT | { "config": {...} }/namespaces/{namespace-id}/configurations/{config-idtype}/
objects/{object-id}/ | content stored as is | 200 OK: config object saved successfully 409 CONFLICT: draftconfig with object-name id already exists 500 Error: while saving the draft | PUT{ "version" : "version-id" } | |||||
POST | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions
| ||||||||
{ "config ": {...} } | content stored as is | 200 OK: config object updated successfully 404 NOT Found : config object doesn't exist already, cannot be updated. 500 Error while updating the config | { "version" : "version-id" } | ||||||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions | 200 return all the versions for the config identified by the configobject-nameid 404 config object not found 500 error while getting config object |
| ||||||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions/{version-number}-1 -> latest version
| 200 return the versions for the config identified by the config-id and version-numberreturns the specific version of the object 404 config object with version found { "timestamp" : "...", "config": { "source" : { .... }, "transforms" : [...], "sinks" [...] "connections" : [..] } }500 error while getting config
object | contents returned as is | ||||||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} Get latest version | 200 return the latest version for the config object 404 config object with version found 500 error while getting the latest config object | content returned as is | ||||||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects | 200 return the name of list of all saved configsmetadata about config objects 500 error | [ "name" : "StreamToTPFS", "DBToHBase""lastSaved": "..", .. } , | ||||||
DELETE | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} | DELETE | /namespaces/{namespace-id}/configurations/{config-id} | 200 successfully deleted the specified config 200 successfully deleted all configs 500 error while deleting | 404 config the specified object 404 object does not exist 500 error while deleting |
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
Code Block | ||
---|---|---|
| ||
GET namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/
// create a new plugin template
POST namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json'
// update existing plugin template
PUT namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json'
// delete the plugin template
DELETE namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ |
Code Block | ||
---|---|---|
| ||
// create/update defaults this include user's plugin version preferences, etc.
PUT : namespaces/{namespace-id}/defaults -d '@default.json'
GET : namespaces/{namespace-id}/defaults |
JAVA API - Config Store:
title | Existing configstore methods |
---|
"Drafts", "Plugin Templates", "Default versions" and "Dashboards" are type of configurations specified as "config-type" in the REST call.
The individual JSON-config or object would be identified by "object-id".
JAVA API - Config Store:
Code Block | ||
---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException;
void createOrUpdate(String namespace, String type, Config config);
void delete(String namespace, String type, String id) throws ConfigNotFoundException;
List<Config> list(String namespace, String type);
Config get(String namespace, String type, String id) throws ConfigNotFoundException;
void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry. Config get(String namespace, String type, Config config); void delete String id, int version) throws ConfigNotFoundException; // get all the versions of an entry. List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException; List<Config> list(String namespace, String type); Config get(String namespace, String type, String id) throws ConfigNotFoundException; void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry.
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException;
// get all the versions of an entry.
Config getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException;
// delete all entries of specified type.
void delete(String namespace, String type) |
Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
3) Dependent UI changes. |
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Code Block |
---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
User Stories (3.5.0)
- For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
- query for plugins available for a certain artifacts and list them in UI
- obtaining output schema of plugins provided the input configuration information
- deploying pipeline and start/stop the pipeline
- query the status of a pipeline run and current status of execution if there are multiple stages.
- get the next schedule of run, ability to query metrics and logs for the pipeline runs.
- creating and saving pipeline drafts
- get the input/output streams/datasets of the pipeline run and list them in UI.
- explore the data of streams/datasets used in the pipeline if they are explorable.
- Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
- delete hydrator pipeline
- the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.
Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.
Generic Endpoints
...