Option #3

Story 1 - Schema and field value suggestions :

Plugin annotation @Endpoint:

Plugin’s can have custom plugin-specific methods that can be annotated as @Endpoint.

UI can learn about available endpoints for a plugin from the plugin properties.

UI can call the app-fabric endpoint identifying {artifact, plugin} with the method name and method-parameter as request body, the app-fabric endpoint will then load the corresponding plugin and call the method identified by method-name, if the method is annotated as @Endpoint.

The response from this method call is sent as response of the HTTP request.

REST API :

Code Block

POST : /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/
{plugin-type}/plugins/{plugin-name}/versions/{plugin- version}/methods/{plugin-method}?scope={artifact-scope}&pluginScope={plugin-scope}
 
Request-Body :  Map<String, String>JSON -  fieldName to value mapping.
 
Response : 
200, Successful Response JSON string

Endpoint Annotation

Code Block
@Retention(RetentionPolicy.RUNTIME) public @interface Endpoint { /** * Returns the endpoint. */ String endpoint(); }

Example Methods in Plugin DBSource:

Code Block

title	DBSource

@Endpoint("listTables")
List<String> listTables(ListTableRequest request)

@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request)

Story 2 - Drafts

HTTP Request Type

Endpoint

Request Body

Response Status

Response Body

PUT

/namespaces/{namespace-id


404, Not Found, Plugin Specific Error Message (Example : DB, Table not found)
500, Error, Plugin Specific Error Message (Example : JDBC Connection error)

Endpoint Annotation

Code Block
@Retention(RetentionPolicy.RUNTIME) public @interface Endpoint { /** * Returns the endpoint. */ String endpoint(); }

Example Methods in Plugin DBSource:

Code Block

title	DBSource

@Endpoint("listTables")
List<String> listTables(ListTableRequest request)

@Endpoint("getSchema")
Map<String, String> getSchema(SchemaRequest request)

Story 2 - Drafts

Configurations HTTP Handler:

Single HTTP Handler for unifying Console Setting Handler and Dashboards.

/namespacesnamespace/configurations/ the name of list of all saved configs

500 error

[ nameStreamToTPFSlastSaved "",

,
...
]

DELETE

HTTP Request Type

Endpoint

Request Body

Response Status

Response Body

PUT

/namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/

{

"config": {...}

}

200 OK: config saved successfully

409 CONFLICT: draftconfig with object-name id already exists

500 Error: while saving the draft

POST

/namespaces/{namespace-id}/configurations{config-type}/objects/{configobject-id}/versions

{

"config ": {...}

}

200 OK: config object updated successfully

404 NOT Found : config object doesn't exist already, cannot be updated.

500 Error while updating the config

GET

/namespaces/{namespace-id}/configurations/{config-idtype}/versions/objects/{object-id}versions

200 return all the versions for the config identified by the configobject-nameid

404 config not found

500 error while getting config

[

{

"timestamp" : "...",

"version" : ".."

},

...

]

GET

/namespaces/{namespace-id}/configurations/{config-type}/objects/{object-id}/versions/{version-number}

200 return the versions for the config object identified by the configobject-id and version-number

404 config object with version found

500 error while getting config object

{

"timestamp" : "...",

"config": {

"source" : {

....

},

"transforms" : [...],

"sinks" [...]

"connections" : [..]

}

GET

/namespaces/{namespace-id}/configurations/{config-idtype}/

Get latest version

GET

objects/{

object-id}

Get latest version

200 return

the latest version for the config object

404 config object with version found

500 error while getting config object

{

"

timestamp" : "

...",

"

config":

{

..

}

GET

/namespaces/{namespace-id}/configurations/{config-idtype}/objects

200 successfully deleted the specified config

404 config does not exist

500 error while deleting

The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.

Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:

Code Block

title	Plugin Template Endpoints

GET

return the name of list of all saved config objects

500 error

[
{

"name" : "StreamToTPFS",

"lastSaved": "..",

..

}

,
...
]

DELETE

/namespaces/{namespace-id}/

plugin-templates

configurations/{

plugin

config-

template-id

type}/

// create a new plugin template POST namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // update existing plugin template PUT namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ -d '@plugin-template.json' // delete the plugin template DELETE namespaces/{namespace-id}/plugin-templates/{plugin-template-id}/ Code Block

title	Defaults

// create/update defaults this include user's plugin version preferences, etc. PUT : namespaces/{namespace-id}/defaults -d '@default.json' GET : namespaces/{namespace-id}/defaults

objects/{object-id}

200 successfully deleted the specified object

404 object does not exist

500 error while deleting

The ConsoleSettingsHttpHandler currently makes use of ConfigStore. It's however not name-spaced and has few other issues, it can be fixed and can be improved to store configs.

Along with pipeline drafts ConsoleSettingsHttpHandler also stores the following information currently:

"Drafts", "Plugin Templates", "Default versions" and "Dashboards" are of "config-type" and the individual json-config or object would be identified by "object-id".

JAVA API - Config Store:

Code Block

title	Existing configstore methods

void create(String namespace, String type, Config config) throws ConfigExistsException;

void createOrUpdate(String namespace, String type, Config config);

void delete(String namespace, String type, String id) throws ConfigNotFoundException;

List<Config> list(String namespace, String type);

Config get(String namespace, String type, String id) throws ConfigNotFoundException; 

void update(String namespace, String type, Config config) throws ConfigNotFoundException;

Code Block

title	Configstore new methods

// get a particular version of an entry. 
Config get(String namespace, String type, String id, int version) throws ConfigNotFoundException; 
// get all the versions of an entry.
List<Config> getAllVersions(String namespace, String type, String id) throws ConfigNotFoundException;

Open Questions :

1) ConfigStore stores the configs in "config.store.table", currently the table properties doesn't have versioning, drafts would need versioning, this would also need CDAP-upgrade to update properties for the existing dataset?

2) rename ConsoleSettingsHttpHandler to ConfigurationsHttpHanlder ?

3) Dependent UI changes.

Schema Propagation and Validation through backend - DryRuns:

Currently when pipeline is published, configurePipeline of plugins are called and we perform pipeline validation and plugin validations and also deploy the application.

1.Goal of dry-run endpoint is to validate a pipeline, then validate plugins by calling configure methods of plugin’s in the pipeline without

performing any creation of datasets or generate program etc, which are usually done during deploy.

2. using dry-run we would be able to catch issues in the pipeline earlier and fix them before deploying.

Dry-run can also be used by UI for schema propagation with some requirements from UI:

If Plugin has field “schema", UI can mutate the output schema

If plugin doesn’t have the field “schema" , UI cannot change the output schema and has to rely on result of dry

run for the output schema for that stage, which is set during plugin configuration.

we need to follow the above conditions for correctness, if UI mutates schema when there isn’t a field “schema”, the backend would have a different schema as input-schema for the next stage and the UI changes wouldn’t be reflected.

Code Block
POST : namespace/{namespace-id}/dry-run Request-Body : JSON Config. Response-Body: JSON Config with additional fields in the plugin for output schema, exceptions in configuring pipeline stage, etc.

User Stories (3.5.0)

For the hydrator use case, the backend app should be able to support hydrator related functionalities listed below:
query for plugins available for a certain artifacts and list them in UI
obtaining output schema of plugins provided the input configuration information
deploying pipeline and start/stop the pipeline
query the status of a pipeline run and current status of execution if there are multiple stages.
get the next schedule of run, ability to query metrics and logs for the pipeline runs.
creating and saving pipeline drafts
get the input/output streams/datasets of the pipeline run and list them in UI.
explore the data of streams/datasets used in the pipeline if they are explorable.
Add new metadata about a pipeline and retrieve metadata by pipeline run,etc.
delete hydrator pipeline
the backend app's functionalities should be limited to hydrator and it shouldn't be like a proxy for CDAP.

Having this abilities will remove the logic in CDAP-UI to make appropriate CDAP REST calls, this encapsulation will simplify UI's interaction with the back-end and also help in debugging potential issues faster. In future, we could have more apps similar to hydrator app so our back-end app should define and implement generic cases that can be used across these apps and it should also allow extensibility to support adding new features.

Generic Endpoints

...

Versions Compared

Old Version 86

New Version 87

Key

Story 1 - Schema and field value suggestions :

Plugin annotation @Endpoint:

REST API :

Story 2 - Drafts

Story 2 - Drafts

Configurations HTTP Handler:

JAVA API - Config Store:

Open Questions :

Schema Propagation and Validation through backend - DryRuns:

User Stories (3.5.0)

Generic Endpoints

Page Comparison

Versions Compared

Old Version 86

New Version 87

Key

Story 1 - Schema and field value suggestions :

Plugin annotation @Endpoint:

REST API :

Story 2 - Drafts

Story 2 - Drafts

Configurations HTTP Handler:

JAVA API - Config Store:

Open Questions :

Schema Propagation and Validation through backend - DryRuns:

User Stories (3.5.0)

Generic Endpoints