Hydrator Plugin Experience

Installing plugins

In CDAP 3.2, installing Hydrator plugins requires uploading the artifact containing the plugins, placing a .json the UI uses to configure widgets into a special folder on the UI machine, and adding docs to cdap's website.

This is problematic for a few reasons:

  1. Too complex.  There are multiple steps required, and they need to happen to multiple systems (against cdap and cdap-ui).

  2. The same UI configs are shared across all plugin artifacts, which is incorrect. Artifacts are versioned, and different versions may require different configs.  Additionally, artifacts with the same name and version may still be different if they are in different namespaces

  3. No way to add docs since they are on the Cask hosted docs page

The problem is that widgets and docs don't belong in CDAP, but they also don't really belong in the UI either. The UI can live on multiple machines, and also don't want to force operations on the filesystem when a user is simply installing a new set of plugins.  Long term, what we really would want is a Hydrator backend, which is an app that manages pipelines, plugins, etc.  In the shorter term, it would be desirable if there were some generic CDAP feature that could allow Hydrator to use CDAP to serve the widget and doc information.

The plan then, is to add metadata to artifacts. Hydrator can store the widgets and docs for each plugin in the metadata of the artifact containing that plugin. We can already add metadata to apps, programs, datasets, streams. It makes sense to have the feature for artifacts as well.  This solves the 3 problems mentioned above, while keeping CDAP generic.

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata
 
GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties
POST /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties
DELETE /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties
DELETE /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties/{property}
 
GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags
POST /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags
DELETE /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags/{tag}

Note: It doesn't look like there is an API for getting a specific property. Not sure why there isn't, but seems to make sense to have one.

Installing a set of plugins could then be a single command:

load artifact <path/to/artifact> [config-file <config-file>] [with metadata <metadata-file>] 

Using and Installing plugins

...