Terminology

...

Mocks

These mocks (not final) give an idea of the flow users will go through in order to install a package from the Cask Market.

The user clicks the '+' button, they will see the list of groups from which they can choose and add a package. For example, 'Hydrator Plugins', or 'Example Applications' could both be groups.package - then selects a 'category' from the left sidebar to list 'packages' that can be installed.

Image Added

When the user chooses a package to install, more information is displayed, including one or more steps involved in installing the package. Each step is a wizard for creating some CDAP entity.

Image Added

Terminology

package - A collection of entities (artifacts, applications, datasets, streams) to add to CDAP. A package is identified by a name and version, and belongs to a groupcan be tagged with one or more categories. A package consists of an archive of resources (tarball) and a package spec.

package spec - A json file containing a list of actions to perform against CDAP. For example, a spec for the Purchase History example will include an action to add the Purchase History artifact, then an action to create an application from that artifact.

package archive - A tarball zip containing any resources needed to perform the actions in the package spec. For example, if the spec contains an action to add an artifact, the archive must contain the jar file to add.

APIs

All APIs are relative to a base path. For example, cask.co/marketplace/v1. In the initial version, there will only be GET APIs.

List Groups

...

category - A package can be tagged with one or more categories. A category corresponds to one of the tabs on the left bar of the mocks.

Architecture

There will be a set of marketplace APIs that the UI will use to get categories and packages. In the first version of the market, the APIs will simply be static content served from S3. This essentially amounts to placing packages in a pre-determined directory structure, and generating a file that lists all packages in the repository.

There will be an internal process to push the entire market repository to S3. If a user wishes to host their own marketplace, they can do so by sticking a server (apache httpd for example) on top of a local directory structure.

APIs

The APIs are simply a contract about the directory structure of the marketplace. All APIs are relative to a base path. For example, cask.co/marketplace/v1. The structure is expected to be:

Code Block

GET
<base>/v1/packages.json
<base>/v1/packages/<package-name>/<version>/icon.png
<base>/v1/packages/<package-name>/<version>/license.txt
<base>/v1/packages/<package-name>/<version>/spec.json
<base>/v1/packages/<package-name>/<version>/spec.json.asc
<base>/v1/packages/<package-name>/<version>/archive.zip
<base>/v1/packages/<package-name>/<version>/archive.zip.asc

The packages.json and signature files could be generated from all the package spec.json files using a tool.

List all Packages

Code Block

GET /v1/<cdap-version>/packages.json
ex: GET /v1/packages.json
[
  {
    "name": "PurchaseExample",
    "descriptionlabel": "Purchase History"Example,
applications to get started with CDAP."
  },
  {"description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "nameauthor": "hydrator-pluginsCask",
    "labelorg": "Hydrator PluginsCask Data Inc.",
    "descriptionversion": "Collections of plugins to extend Hydrator functionality."
  },
  ...
]

Get Group Icon

Code Block
GET /groups/<group>/icon ex: GET /groups/examples/icon [binary image contents]

List Packages in a Group

Code Block

GET /groups/<group>/packages
ex: GET /groups/examples/packages
[
  {
    "name": "PurchaseExample4.0.1",
    "categories": [ "examples" ],
    "cdapVersion": "[4.0.0,4.1.0)
  },
  {
    "name": "HelloWorld",
    "label": "PurchaseHello HistoryWorld",
    "description": "ExampleSimple Applicationapplication demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },

 {     "nameversion": "HelloWorld4.0.0",
    "labelcategories": [ "examples"Hello World"],
    "descriptioncdapVersion": "Simple application demonstrating usage of flows and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  [4.0.0,4.1.0)"
  },
  ...
]

Get Package Icon

Code Block
GET /groups/<group>/packages/<package>/icon ex: GET /groups/examples/packages/PurchaseExample/icon [binary image contents]

List Package Versions

Code Block

GET /groups/<group>/packages/<package>/versions
ex: GET /groups/examples/packages/PurchaseExample/versions
[
  {    
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.1",
    "created": 1234567899,
    "changelog": [
      "fixed a small parsing bug"
    ]
  },
  {    
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.0",
    "created": 1234567890,
    "changelog": [
      "updated APIs to work with CDAP 4.0.0"
    ]
  },
  ...
]

Get Package Archive

Code Block
GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz [ binary archive contents]

Get Package Archive Signature

Code Block
GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz.asc ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz.asc [ archive signature ]

Get Package Spec

...

This list is not expected to change often. It can be cached by the UI if needed. The 'cdapVersion' specifies which versions of cdap the package is compatible with. If none is given, it is compatible with all versions.

Note

This leaves grouping by category up to the UI. If needed, we could perhaps add packages-<category>.json files that only list the packages in a specific category.

This also leaves display of multiple versions of the same package up to the UI. Though it seems like most of the time we would only have one version of the package per cdap version so maybe it's not a big problem.

This also leaves filtering of packages incompatible with the cdap instance up to the UI.

Get Package Archive

Code Block
GET /v1/packages/<package-name>/<version>/archive.zip ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip [ binary archive contents]

Get Package Archive Signature

Code Block
GET /v1/packages/<package-name>/<version>/archive.zip.asc ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip.asc [ archive signature ]

Get Package Spec

Code Block

GET /v1/packages/<package-name>/<version>/spec.json
ex: GET /v1/packages/PurchaseExample/4.0.0/spec.json
{
  "specVersion": "1.0",
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.0",
  "created": 1234567899,
  "cdapVersion": "[4.0.0,4.1.0)",
  "changelog": "fixed a small parsing bug",
  "categories": [ "examples" ],
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "name",
          "value": "PurchaseHistoryExample"
        },
        {
          "name": "version",
          "value": "4.0.1"
        },
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "jar",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "name",
          "valuedefault": "PurchaseHistoryExamplePurchaseHistory"
        },
      ]
 {   }
  ]
}

Get Package Spec Signature

Code Block
GET "name": "version", "value": "4.0.1" }, { "name": "scope", "value": "user" }, { /v1/packages/<package-name>/<version>/spec.asc ex: GET /v1/packages/PurchaseExample/4.0.0/spec.asc [ spec signature ]

Get Package Icon

Code Block
GET /v1/packages/<package-name>/<version>/icon.png ex: GET /v1/packages/PurchaseExample/4.0.0/icon.png [ icon bytes ]

Get Package License

Code Block

GET /v1/packages/<package-name>/<version>/license.txt
ex: GET /v1/packages/PurchaseExample/4.0.0/license.txt
Copyright © 2014-2016 Cask Data, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
        "name": "jar",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "name",
          "default": "PurchaseHistory"
        }
      ]
    }
  ],
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    }
  }
}

Get Package Spec Signature

Code Block
GET /groups/<group>/packages/<package>/versions/<version>/spec.asc ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/spec.asc [ spec signature ]

Security

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package.

Package Spec

The package spec contains some metadata about the spec itself, and a list of actions to perform on the CDAP instance. It is a JSON file of the following structure:

Code Block

{
  "metadata": {
    "spec-version": "1.0"
  },
  "actions": [
    actionspec1,
    actionspec2,
   http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
...

Security

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package. There will also be a setting that lets people turn off signature checking in case its not needed for internally hosted repositories.

Package Spec

The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:

Code Block

{
  "specVersion": "1.0"
  "name": "<name>",
  "version": "<version>",
  "label": "<label>",
  "description": "<description>",
  "org": "<org>",
  "categories": [ <categories> ],
  "cdapVersion": "<compatible-versions>",
  "changelog": "<changes>",
  "actions": [
    actionspec1,
    actionspec2,
    ...
  ]
}

The actions in the spec will correspond to steps in the UI wizard for installing the package.

...

Each action will contain a type, a list of arguments, and dependencies. Each type of action will require different arguments. In the first version, the following types will be supported: create_artifact, create_app, create_stream, create_dataset, create_hydrator_draft.

Code Block

{
  "type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft",load_datapack" | "install_package"
  "arguments": [
    {
      "name": [argument name],
      "value": [argument value],
      "canModify": true | false (defaults to false)
    }
  ]
}

Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.

...

name	description	required?	default
name	artifact name	yes
jar	name of jar file in package archive	yesno (if using externalArchive)	archivelink
externalJar	link to download 3rd party jar	no	none
externalArchive	link to download 3rd party archive	no	none
archivesigexternalArchiveSignature	link to get 3rd party archive signature	no	none
scope	artifact scope (externalArchiveJar	path of the jar file in the external archive	no	none
scope	artifact scope (implies API to add system artifacts is added in 4.0)	no	user
version	artifact version to pass as Artifact-Version header	no	none
parentsconfig	config file contains artifact parents to pass as Artifact-Extends header	no	none	plugins	artifact plugins to pass as Artifact-Plugins header, plugins, and properties	no	none

create_app

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#create-an-application

...

name	description	required?	default
name	stream name	yes
description	stream description, results in call to set stream properties	no	empty
format	stream format as json object, results in call to set stream properties	no	empty
schema	stream schema, results in call to set stream properties	no	empty
ttl	stream ttl, results in call to set stream properties	no	empty
notification.threshold.mb	mb threshold for sending notifications, results in call to set stream properties	no	empty	loadfiles	files in the package archive to write to the stream. results in a call to write to the stream in batch	no	empty

create_dataset

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/dataset.html#creating-a-dataset

name	description	required?	default
name	dataset name	yes
type	dataset type	yes
description	dataset description	no	empty
properties	json map of dataset properties	no	empty

...

load_

...

datapack

Loads a datapack into some dataset or stream.

scope, name, version of the artifact to create the app withpipeline config (file in the package archive)

name	description	required?	default
namepipeline	dataset/stream name	yes		artifact
files	files to load into the dataset/stream	yes		config

install_package

Installs another package from the marketplace.

name	description	required?	default
name	package name	yes

Dependencies

Packages will only be able to specify dependencies on the CDAP version, as well as dependencies on the existence of specific CDAP entities. For example, the core-plugins-1.5.0 package requires that there exist system artifacts cdap-data-pipeline-4.0.0 and cdap-data-streams-4.0.0 in the CDAP instance.

Code Block

{
  ...
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    },
    "artifacts": [
      { 
        "scope": "system",
        "name": "spark-plugins",
        "minVersion": "1.5.0",
        "maxVersion": "1.6.0"
      },
      ...
    ],
    "streams": [
      { "name": "smsTexts" }
    ],
    "datasets": [
      { "name": "spamTexts" }
    ]
  }
}

Min versions are inclusive and max versions are exclusive.

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.

Architecture

There will be a set of marketplace APIs that the UI will use to get groups, packages, package versions, icons, and package tarballs. There will be a market server that powers these APIs. The server will use a set of internal storage interfaces that define how to read the information required by the APIs. We can start with a storage implementation that simply reads from local files, and perhaps another storage implementation that reads from cloud storage like S3.

Image Removed

The market server will be stateless, so a load balancer can be placed in front of it to ensure that it is highly available and to ensure that it can handle a high volume of requests

Image Removed

File Store

The first implementation of the storage layer can simply be a store that looks at a filesystem for files containing the relevant information. The File Store will expect a specific directory structure:

Code Block

<base dir>/<group>/icon.jpg
<base dir>/<group>/meta.json
<base dir>/<group>/<package>/<version>/spec.json
<base dir>/<group>/<package>/<version>/icon.jpg
<base dir>/<group>/<package>/<version>/archive.tgz
 
ex:
/opt/cdap/marketplace/examples/icon.jpg
/opt/cdap/marketplace/examples/meta.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/icon.jpg
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/icon.jpg

On start up, the server will scan the base directory, load relevant information into memory, and simply serve data based on the contents of the files. This would also let ops teams manage the marketplace through use of 'group' packages and 'cask package' packages.

Note: with such little logic in the server, why bother having a server at all? Why not just stick an Apache server in front of files or serve directly from S3? The assumption is that we will need to add more complicated functionality in the future, such as APIs to add groups and packages, ability to search for packages by various fields, etc.

Example Use Cases

Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline

When the user clicks on the '+' button, the UI makes a call:

Code Block

GET /groups
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydrator-pipelines",
    "label": "Hydrator Pipelines",
    "description": "Templates of various Hydrator pipelines."
  },
  ...
]

to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Pipelines', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Pipelines' group:

Code Block

GET /groups/hydrator-plugins/packages
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  ...
]

Among that list is the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get all versions of that package:

Code Block

GET /groups/hydrator-plugins/packages/sfdc-lead-dump/versions
[
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.1",
    "created": 1234567899,
    "changelog": [
      "fixed a small parsing bug"
    ],
    "dependencies": {
      "cdap": {
        "minVersion": "4.0.0",
        "maxVersion": "4.1.0"
      }
    }
  },
  ...
]

It defaults to the most recent version that is compatible with the version of CDAP that is running. The user decides to install the package, so the UI makes a call to get the package spec:

Code Block

GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec
{
  "name": "sfdc-lead-dump",
  "label": "SFDC Lead Dump",
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "1.0.1",
  "created": 1234567899,
  "changelog": [
    "fixed a small parsing bug"
  ],
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user",
          "canModify": false
        },
        {
          "name": "name",
          "value": "sfdc-plugins",
          "canModify": false
        },
        {
          "name": "version",
          "value": "1.0.0",
          "canModify": false
        },
        {
          "name": "parents",
          "value": "system:cdap-data-pipeline[4.0.0,4.1.0)",
          "canModify": false
        },
        {
          "name": "jar",
          "value": "sfdc-plugins.jar", // file in the archive
          "canModify": false
        }
      ]
    },
    {
      "type": "create_hydrator_draft",
      "arguments": [
        {
          "name": "artifact",
          "value": {
            "scope": "system",
            "name": "cdap-data-pipeline",
            "version": "4.0.0"
          },
          "canModify": false
        },
        {
          "name": "name",
          "value": "SFDC Lead Dump",
          "canModify": true
        },
        {
          "name": "config",
          "value": "sfdc.json", // file in the archive
          "canModify": false
        }
      ]
    }
  ]
}

The UI also fetches the spec signature and uses the public key to validate the spec:

Code Block
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec.asc

The UI also fetch the package archive and signature. It validates the package, and writes the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator draft.

Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.

When the user clicks on the '+' button, the UI makes a call:

...

version

package version

yes

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? We will not attempt rollback or anything like that. Instead, all the wizards that execute the actions must be idempotent. For example, if told to add an artifact and the artifact already exists, the step can simply be skipped.

Hosting a Custom Marketplace

To host a custom marketplace, users can run an apache httpd server on top of a local directory structure. To make this easier, we could create a github repository of all the public packages hosted by Cask. The repository will follow the directory structure documented here, and have a script at the top level that builds the zip, signs the zips and specs, and generates the packages.json file.

Example Use Cases

Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline

When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:

Code Block

GET /v1/packages.json
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.0",
    "categories": [ "hydrator-pipelines" ]
  },
  ...
]

Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.0/license.txt [ apache2 license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /v1/packages/sfdc-lead-dump/1.0.0/spec.json
{
  "name": "sfdc-lead-dump",
  "label": "SFDC Lead Dump",
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "1.0.1",
  "created": 1234567899,
  "changelog": "",
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "examplesscope",
    "label": "Examples",     "descriptionvalue": "Exampleuser"
applications to get started with CDAP."   },
        {
          "name": "name",
          "value": "hydratorsfdc-plugins"
        },
    "label": "Hydrator Plugins",    {
          "descriptionname": "Plugins for Hydrator Pipelines."
  }version",
  ... ]

to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Plugins', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Plugins' group:

Code Block

GET /groups/hydrator-plugins/packages
[
  ...,
  {       "value": "1.0.0"
        },
        {
          "name": "mysql-jdbc-driverconfig",
          "labelvalue": "MySQL JDBC Driver",sfdc-plugins.json" // file in the archive
        },
        {
"description": "JDBC Driver for MySQL databases.",     "authorname": "MySQLjar",
          "orgvalue": "Oracle"
  },sfdc-plugins.jar" // file in the archive
    ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get all versions of that package:

Code Block

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions
[
  {
    }
      ]
   "name": "mysql-jdbc-driver", },
    {
      "labeltype": "MySQLcreate_app",
JDBC Driver",     "descriptionarguments": "JDBC Driver for MySQL databases.", [
        {
    "author": "MySQL",     "orgname": "Oracleartifact",
    "version": "5.1.39",     "createdvalue": 1234567899,{
     "changelog": [ ],     "dependenciesscope": { }"system",
        },   ...
]

The user decides to install the 5.1.38 version of the driver. The UI makes a call to get the spec, and to get the spec signature to make sure it is valid:

Code Block

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec
{ "name": "cdap-data-pipeline",
            "version": "4.0.0"
          }
"name": "mysql-jdbc-driver",     "label": "MySQL JDBC Driver"},
  "description": "JDBC Driver for MySQL databases.",
  "author": "MySQL",      {
          "orgname": "Oraclename",
  "version": "5.1.39",   "created": 1234567899,   "actionsvalue": [
"SFDC Lead Dump",
   {       "typecanModify": "create_artifact",true
       "arguments": [ },
        {
          "name": "scopeconfig",
          "value": "usersfdc.json", // file in the archive
     "canModify":  false }
      ]
},    }
  ]
 {
          "name": "name",
          "value": "mysql-connector-java",
          "canModify": false
        },
        {
     }

The UI also gets the spec signature to validate the spec:

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.1/spec.json.asc

The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator pipeline.

Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.

When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:

Code Block

GET /v1/packages.json
[
  ...,
  {
    "name": "versionmysql-jdbc-driver",
          "value"label": "5.1.39MySQL JDBC Driver",
          "canModify"description": false"JDBC Driver for       }MySQL databases.",
    "author": "MySQL",
  {  "org": "Oracle",
       "nameversion": "parents5.1.39",
     
    "valuecategories": [ "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]",
          "canModify": false
        },
  hydrator-plugins" ]
  },
  ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /v1/packages/mysql-jdbc-driver/5.1.39/license.txt [ gpl license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /v1/packages/mysql-jdbc-driver/5.1.39/spec.json
{    
 { "name": "mysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
  "namedescription": "jarJDBC Driver for MySQL databases.",
  "author": "MySQL",
  "org": "Oracle",
  "valueversion": "mysql-connector-java-5.1.39-bin.jar",
// file in the archive
   "categories": [ "hydrator-plugins" ]
  "created": 1234567899,
  "canModifyactions": false[
    {
   },   "type": "create_artifact",
    {  "arguments": [
       "name": "archivelink" {
          "valuename": "https://dev.mysql.com/downloads/file/?id=462849scope",
          "canModifyvalue": false"user"
        },
        {
          "name": "archivesigname",
          "value": "https://dev.mysql.com/downloads/gpg/?file=  "value": "mysql-connector-java-5.1.39.tar.gz"
        },
        {
          "name": "pluginsversion",
          "value": { "5.1.39"
        },
        {
          "parentsname": [ "cdap-data-pipeline[3.0.0,10.0.0]" ],
 externalArchive"
          "pluginsvalue": ["https://dev.mysql.com/downloads/file/?id=462849"
        },
     {   {
             "name" : "mysqlexternalArchiveSignature",
          "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz"
        "type"},
: "jdbc",       {
          "classNamename" : "com.mysql.jdbc.DriverexternalArchiveJar",
  
             "descriptionvalue" : "Plugin for MySQL JDBC drivermysql-connector-java-5.1.39-bin.jar"
        },
     }   {
         ] "name": "config",
        },  "value": "mysql-connector-java-5.1.39.json" // file in the archive containing parents "canModify": falseand plugins
        }
      ]
    }
  ]
}

The UI also makes a call to get the spec signature to make sure it is valid:

Code Block
GET /v1/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc

The UI then makes calls to get the archive and its signature to validate the archive, and unpack unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.

Code Block
GET /groups/hydrator-pluginsv1/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip.asc GET /groupsv1/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip

Page Comparison

Versions Compared

Old Version 21

New Version Current

Key

Table of Contents

Terminology

Mocks

Terminology

APIs

List Groups

Architecture

APIs

List all Packages

Get Group Icon

List Packages in a Group

Get Package Icon

List Package Versions

Get Package Archive

Get Package Archive Signature

Get Package Spec

Get Package Archive

Get Package Archive Signature

Get Package Spec

Get Package Spec Signature

Get Package Icon

Get Package License

Get Package Spec Signature

Security

Package Spec

Security

Package Spec

create_app

create_dataset

load_

datapack

install_package

Dependencies

Failures

Architecture

File Store

Example Use Cases

Failures

Hosting a Custom Marketplace

Example Use Cases