Hydrator Plugin Experience

Installing custom plugins

In CDAP 3.2, installing Hydrator plugins requires uploading the artifact containing the plugins, placing a .json the UI uses to configure widgets into a special folder on the UI machine, and adding docs to cdap's website.

...

The plan then, is to add metadata properties to artifacts. Hydrator can store the widgets and docs for each plugin in the metadata properties of the artifact containing that plugin. We You can already add metadata to apps, programs, datasets, streams. It makes sense to have the feature for artifacts as well.  This set properties on streams and datasets, this would be an analogous feature. This solves the 3 problems mentioned above, while keeping CDAP genericignorant of widgets and UI specific things.

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadataproperties
 
GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties
POSTproperties/{property}?keys=key1,key2
PUT /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadataproperties/properties{property}
DELETEPUT /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties
DELETE /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/properties/{property}
 
GET /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags
POST /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags
DELETE /namespaces/{namespace-id}/artifacts/{artifact-name}/{versions}/{artifact-version}/metadata/tags/{tag}

Note: It doesn't look like there is an API for getting a specific property. Not sure why there isn't, but seems to make sense to have one.

Installing a set of plugins could then be a single command:

load artifact <path/to/artifact> [config-file <config-file>] [with metadata <metadata-file>] 

...

Installing a set of plugins could then be a single command:

load artifact <path/to/artifact> [config-file <config-file>] 

with properties added to the config file, and something of the form:

{
  "properties": {
    "widgets.batchsource.database": "<widgets json>",
    "doc.batchsource.database": "<doc>",
    ...
  }
}

Installing plugins from hydrator-plugins repository

To install the plugins from hydrator-plugins, a user simply installs the hydrator-plugins rpm or deb.  In addition, cdap-site.xml will have to set the app.artifact.dir setting to include the directory for those plugins (/opt/hydrator-plugins/artifacts).

The work for this will include setting up the hydrator-plugins build to create the rpms and debs.  The pom will be changed so that in addition to the jars, the config json will be created.  The json files will be created using an antrun script target that writes out the parents for the artifact, as well as properties for widget and doc information.  Prepare phases can then copy jars and jsons to the right place for packaging to be done.

The CDAP build will still need to bundle the results of the hydrator-plugins build in order to build a standalone zip that contains hydrator plugins. To accomplish this, we will add a property ('-Dadditional.artifacts.dir=</path/to/additional/artifacts>') that can be used to copy any additional jars and jsons from some external directory. This property can then be used in standalone builds to pull in the hydrator-plugins artifacts and configs.

Plugin Selection

User installs CDAP 3.3.0 by downloading the sdk or installing via packages.  Included in CDAP is the Hydrator webapp, cdap-etl-batch-3.3.0, and cdap-etl-realtime-3.3.0 artifacts.  Also included are all the v1.0.0 plugins in the hydrator-plugins repo, like cassandra-1.0.0.  The user creates a pipeline that reads from cassandra and writes to a Table.

...