...
Option #1
Description
The hydrator app needs to be able to write/read to a dataset to store and retrieve drafts and other information about business logic. We can implement a Hydrator CDAP Application with a service that can have REST endpoints to serve the required hydrator functionalities. Enabling Hydrator in a namespace will deploy this Hydrator app and start the service. Hydrator UI would ping for this service to be available before coming up. The back-end business logic actions which directly needs to use the CDAP services endpoints can be made generic.
Pros
- Everything (Drafts, etc) stored in the same namespace, proper cleanup when namespace is deleted.
Cons
- Every namespace will have an extra app for supporting hydrator if hydrator is enabled. Running this service, will run 2 containers per namespace. we can add an option to enable/disable hydrator if we are not using hydrator in a namespace. It might feel weird as a user app, as the user didn't write/create this app.
Option #2
Description
We will still use an Hydrator CDAP app but we create an "Extensions" namespace and have the "hydrator" app only deployed in the "extensions" namespace, this app would serve the hydrator requests for all namespaces.
Pros
- Less amount of resources used, only 2 container's used rather than 2 container’s per namespace, only one dataset is used.
- Only one app for using hydrator across namespace and not an app per namespace, less clutter.
- New extensions could be added to the same namespace to support other use cases in future.
Cons
- Using a single dataset for storing all drafts across namespace is less secure?.
- User won't be able to create a new namespace called "Extensions", as it will be reserved.
Open Questions
- How to delete the drafts when the namespace is deleted ?
- When to stop this service?
- Availability of the service?
- Security
- If we decide to add more capability in hydrator back-end app, Eg: Make the pipeline validation/deploy app, etc, then in secure environment,
- The hydrator-service can discover appropriate cdap.service and call appropriate endpoints?
Option #3 (based on discussion with terence)
1) No new user level apps are deployed. Config store is used to store user drafts of hydrator apps.
2) REST endpoint 'configure', can accept partial config and return a config response with suggestions of values for fields in a plugin, exceptions if any during configuring the plugin.
- user can choose a value from the suggestions for the field and call the configure again.
- user can look at exception, fix the issue with either the script or configuration and call configure again.
- when all the required configs are provided and there aren't any exceptions, completionStatus would be set to true for the plugin.
Story 1 - Drafts
HTTP Request Type
Endpoint
Request Body
Response Status
Response Body
POST
#3
Story 1 - Schema and field value suggestions :
Jira Legacy server Cask Community Issue Tracker serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-5149
Plugin annotation @Endpoint:
REST API :
Code Block |
---|
POST : /namespaces/{namespace-id}/ |
artifacts/{ |
artifact- |
name}/ |
{
"config": {...}
}
200 OK: config saved successfully
409 CONFLICT: draft-name already exists
500 Error: while saving the draft
PUT
/namespaces/{namespace-id}/configurations/{config-id}/
{
"config ": {...}
}
200 OK: config updated successfully
404 NOT Found : config doesn't exist already, cannot be updated.
500 Error while updating the config
GET
/namespaces/{namespace-id}/configurations/{config-id}/
200 return all the versions for the config identified by the config-name
404 config not found
500 error while getting config
[
{
"timestamp" : "...",
"config": {
"source" : {
....
},
"transforms" : [...],
"sinks" [...]
"connections" : [..]
}
},
...
]
GET
/namespaces/{namespace-id}/configurations/{config-id}/versions/{version-number}
-1 -> latest version
200 return the versions for the config identified by the config-id and version-number
404 config with version found
500 error while getting config
{
"timestamp" : "...",
"config": {
"source" : {
....
},
"transforms" : [...],
"sinks" [...]
"connections" : [..]
}
}
GET
/namespaces/{namespace-id}/configurations/
200 return the name of list of all saved configs
500 error
[
"streamToTPFS",
"DBToHBase",
...
]
versions/{artifact-version}/types/{plugin-type}
plugins/{plugin-name}/methods/{plugin-method}?scope={artifact-scope}
Request-Body : JSON - fieldName to value mapping.
Response :
200, Successful Response JSON string
404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)
Description : In the request we refer to the plugin-artifact and not the parent artifact. we could use one of the available parent artifact. |
Code Block |
---|
@Retention(RetentionPolicy.RUNTIME)
public @interface Endpoint {
/**
* Returns the endpoint.
*/
String endpoint();
} |
Code Block | ||
---|---|---|
| ||
@Endpoint("listTables")
List<String> listTables(ListTableRequest request)
@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request) |
Story 2 - Drafts
Jira Legacy server Cask Community Issue Tracker serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-5154
Configurations HTTP Handler:
Single HTTP Handler for unifying Console Setting Handler and Dashboards HTTP Handler.
HTTP Request Type | Endpoint : (Table Assumes we are using config-type -> drafts) | Request Body | Response Status | Response Body | |||||
PUT | /namespaces/{namespace-id}/configurations/ | 200 successfully deleted all configs 500 error while deleting | DELETE | /namespaces/{namespace-id}/configurations/{config{config-type}/objects/{object-id}/ | content stored as is | 200 successfully deleted the specified config 404 config does not exist 500 error while deleting |
The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.
Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:
title | Plugin Template Endpoints |
---|
OK: config object saved successfully 409 CONFLICT: config with object-id already exists 500 Error: while saving the draft | { "version" : "version-id" } | |||
POST | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions | content stored as is | 200 OK: config object updated successfully 404 NOT Found : config object doesn't exist already, cannot be updated. 500 Error while updating the config | { "version" : "version-id" } |
GET | /namespaces/{namespace-id}/ |
configurations/{ |
config- |
type}/ |
objects/{object-id}/ |
versions | 200 return all the versions for the config identified by the object-id 404 config object not found 500 error while getting config object |
| ||
GET | /namespaces/{namespace-id}/ |
configurations/{ |
config-type}/objects/{object-id}/ |
Code Block | ||
---|---|---|
| ||
// create/update defaults this include user's plugin version preferences, etc.
PUT : namespaces/{namespace-id}/defaults -d '@default.json'
GET : namespaces/{namespace-id}/defaults |
Config Store:
Code Block | ||
---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException;
void createOrUpdate(String namespace, String type, Config config);
void delete(String namespace, String type, String id) throws ConfigNotFoundException;
List<Config> list(String namespace, String type);
Config get(String namespace, String type, String id) throws ConfigNotFoundException;
void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry.
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException;
// get all the versions of an entry.
Config getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException;
// delete all entries of specified type.
void delete(String namespace, String type) |
Open Questions :
1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?
2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?
Story 2 - Schema and field value suggestions :
REST API:
Request-Method : POST
Request-Endpoint : /namespaces/{namespace-id}/apps/{app-id}/configure
Request-Body
Code Block | ||
---|---|---|
| ||
{
"artifact": {
"name": "cdap-etl-batch",
"scope": "SYSTEM",
"version": "3.4.0-SNAPSHOT"
},
"name": "pipeline",
"config": {
"source": {
"name": "Stream",
"plugin": {
"name": "StreamSource",
"artifact": {
"name": "core-plugins",
"version": "1.3.0-SNAPSHOT",
"scope": "SYSTEM"
},
"properties": {
"format": "syslog",
"name": "test",
"duration": "1d"
}
}
},
"sinks" : [{..}],
"transform": [{..}, {...}]
}
} |
Response-Body
Code Block | ||
---|---|---|
| ||
{
"artifact": {
"name": "cdap-etl-batch",
"scope": "SYSTEM",
"version": "3.4.0-SNAPSHOT"
},
"name": "pipeline",
"config": {
"source": {
"name": "Stream",
"plugin": {
"name": "StreamSource",
"artifact": {
"name": "core-plugins",
"version": "1.3.0-SNAPSHOT",
"scope": "SYSTEM"
},
"properties": {
"format": "syslog",
"name": "test",
"duration": "1d",
"suggestions" : [{
"schema" : [
{
"ts" : "long",
"headers", "Map<String, String>",
"program", "string",
"message": "string",
"pid": "string"
}
]
}],
"isComplete" : "false"
}
}
},
"sinks" : [{..}],
"transform": [{..}, {...}]
}
} |
PipelineConfigurable API Change
Code Block | ||
---|---|---|
| ||
@Beta
public interface PipelineConfigurable {
// change in return-type.
ConfigResponse configurePipeline(PipelineConfigurer pipelineConfigurer) throws IllegalArgumentException;
} |
Code Block | ||
---|---|---|
| ||
public class ConfigResponse {
// list of suggestions for fields.
List<Suggestion> suggestions;
// if there were any exception while executing configure
@Nullable
String exception;
// is the stage configuration complete ?
@DefaultValue("false")
boolean isComplete;
} |
Code Block | ||
---|---|---|
| ||
public class Suggestion {
String fieldName;
// list of possible values for the fieldName
List<String> fieldValues;
} |
Code Block | ||
---|---|---|
| ||
@Beta
public interface ApplicationContext<T extends Config> {
// existing
T getConfig();
// application will set a config response
void setResponseConfig(T response);
// get the response config
T getResponseConfig();
} |
Open Questions:
1) would having setResponseConfig and getResponseConfig ApplicationContext along with input config, allow CDAP programs to set a config and read from other programs, would that be an issue?
2) Database's have information schema table, which has metadata information about column names and their types of tables.
versions/{version-number}
| 200 returns the specific version of the object 404 config object with version found 500 error while getting config object | contents returned as is | ||
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} Get latest version | 200 return the latest version for the config object 404 config object with version found 500 error while getting the latest config object | content returned as is | |
GET | /namespaces/{namespace-id}/configurations/{config-type}/objects | 200 return the list of metadata about config objects 500 error | [ "name" : "StreamToTPFS", "lastSaved": "..", .. } , | |
DELETE | /namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id} | 200 successfully deleted the specified object 404 object does not exist 500 error while deleting |
"Drafts", "Plugin Templates", "Default versions" and "Dashboards" are type of configurations specified as "config-type" in the REST call.
The individual JSON-config or object would be identified by "object-id".
JAVA API - Config Store:
Code Block | ||
---|---|---|
| ||
void create(String namespace, String type, Config config) throws ConfigExistsException;
void createOrUpdate(String namespace, String type, Config config);
void delete(String namespace, String type, String id) throws ConfigNotFoundException;
List<Config> list(String namespace, String type);
Config get(String namespace, String type, String id) throws ConfigNotFoundException;
void update(String namespace, String type, Config config) throws ConfigNotFoundException; |
Code Block | ||
---|---|---|
| ||
// get a particular version of an entry.
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException;
// get all the versions of an entry.
List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException;
|
Schema Propagation and Validation through backend - DryRuns:
- If Plugin has field “schema", UI can mutate the output schema
- If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry
Code Block |
---|
POST : namespace/{namespace-id}/dry-run
Request-Body : JSON Config.
Response-Body:
JSON Config with additional fields in the plugin for output schema,
exceptions in configuring pipeline stage, etc. |
User Stories (3.5.0)
- For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
- query for plugins available for a certain artifacts and list them in UI
- obtaining output schema of plugins provided the input configuration information
- deploying pipeline and start/stop the pipeline
- query the status of a pipeline run and current status of execution if there are multiple stages.
- get the next schedule of run, ability to query metrics and logs for the pipeline runs.
- creating and saving pipeline drafts
- get the input/output streams/datasets of the pipeline run and list them in UI.
- explore the data of streams/datasets used in the pipeline if they are explorable.
- Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
- delete hydrator pipeline
- the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.
Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.
Generic Endpoints
...