...

The APIs are simply a contract about the directory structure of the marketplace. All APIs are relative to a base path. For example, cask.co/marketplace/v1. The structure is expected to be:

Code Block

GET
<base>/<cdap-version>/packages.json
<base>/<cdap-version>/packages/<package-name>/<version>/icon.jpg
<base>/<cdap-version>/packages/<package-name>/<version>/license.txt
<base>/<cdap-version>/packages/<package-name>/<version>/spec.json
<base>/<cdap-version>/packages/<package-name>/<version>/spec.json.asc
<base>/<cdap-version>/packages/<package-name>/<version>/archive.zip
<base>/<cdap-version>/packages/<package-name>/<version>/archive.zip.asc

...

Code Block

GET /<cdap-version>/packages.json
ex: /4.0.0 GET /packages.json
[
  {
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "4.0.1",
    "categories": [ "examples" ],
    "cdapVersion": "[4.0.0,4.1.0)
  },
  {
    "name": "HelloWorld",
    "label": "Hello World",
    "description": "Simple application demonstrating usage of flows and services.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "4.0.0",
    "categories": [ "examples" ],
  },  "cdapVersion": "[4.0..
0,4.1.0)"
  },
  ...
]

This list is not expected to change often. It can be cached by the UI if needed. The 'cdapVersion' specifies which versions of cdap the package is compatible with. If none is given, it is compatible with all versions.

Note

This leaves grouping by category up to the UI. If needed, we could perhaps add packages-<category>.json files that only list the packages in a specific category.

This also leaves display of multiple versions of the same package up to the UI. Though it seems like most of the time we would only have one version of the package per cdap version so maybe it's not a big problem.

This also leaves filtering of packages incompatible with the cdap instance up to the UI.

Get Package Archive

Code Block
GET /<cdap-version>/packages/<package-name>/<version>/archive.zip ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip [ binary archive contents]

Note
If the UI can't download and expand a zip, we'll just have to have all the files here instead of a zip

Get Package Archive Signature

...

Get Package Archive Signature

Code Block
GET /packages/<package-name>/<version>/archive.zip.asc ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip.asc [ archive signature ]

Get Package Spec

Code Block

GET /<cdap-version>/packages/<package-name>/<version>/spec.json
ex: GET /4.0.0/packages/PurchaseExample/4.0.0/spec.json
{
  "spec-version": "1.0",
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.0",
  "created": 1234567899,
  "changelog": "fixed a small parsing bug",
  "categories": [ "examples" ],
  "dependencies": { },
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "name",
          "value": "PurchaseHistoryExample"
        },
        {
          "name": "version",
          "value": "4.0.1"
        },
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "jar",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "name",
          "default": "PurchaseHistory"
        }
      ]
    }
  ]
}

Get Package Spec Signature

Code Block
GET /<cdap-version>/packages/<package-name>/<version>/spec.asc ex: GET /4.0.0/packages/PurchaseExample/4.0.0/spec.asc [ spec signature ]

Get Package Icon

Code Block
GET /<cdap-version>/packages/<package-name>/<version>/icon.jpg ex: GET /4.0.0/packages/PurchaseExample/4.0.0/icon.jpg [ icon bytes ]

Get Package License

Code Block

GET /<cdap-version>/packages/<package-name>/<version>/license.txt
ex: GET /4.0.0/packages/PurchaseExample/4.0.0/license.txt
Copyright © 2014-2016 Cask Data, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
       http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
...

...

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package. There will also be a setting that lets people turn off signature checking in case its not needed for internally hosted repositories.

Package Spec

The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:

...

name	description	required?	default
name	dataset name	yes
type	dataset type	yes
description	dataset description	no	empty
properties	json map of dataset properties	no	empty

Dependencies

Packages will be able to specify dependencies on other packages.

Code Block
{ ... "dependencies": [ { "name": "spark-plugins", "version": "1.5.0" } ] }

If other packages are listed as a dependency, the actions for the dependent package must be executed before the actions of the current package

...

install_package

Installs another package from the marketplace. May not be implemented for version 1.

name	description	required?	default
name	package name	yes
version	package version	yes

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retryWe will not attempt rollback or anything like that. Instead, all the wizards that execute the actions must be idempotent. For example, if told to add an artifact and the artifact already exists, the step can simply be skipped.

Hosting a Custom Marketplace

...

When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:

Code Block

GET /4.0.0/packages.json
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.0",
    "categories": [ "hydrator-pipelines" ]
  },
  ...
]

Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /4.0.0/packages/sfdc-lead-dump/1.0.0/license.txt [ apache2 license ]

...

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /4.0.0/packages/sfdc-lead-dump/1.0.0/spec.json
{
  "name": "sfdc-lead-dump",
  "label": "SFDC Lead Dump",
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "1.0.1",
  "created": 1234567899,
  "changelog": "",
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "name",
          "value": "sfdc-plugins"
        },
        {
          "name": "version",
          "value": "1.0.0"
        },
        {
          "name": "config",
          "value": "sfdc-plugins.json" // file in the archive
        },
        {
          "name": "jar",
          "value": "sfdc-plugins.jar" // file in the archive
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "artifact",
          "value": {
            "scope": "system",
            "name": "cdap-data-pipeline",
            "version": "4.0.0"
          }
        },
        {
          "name": "name",
          "value": "SFDC Lead Dump",
          "canModify": true
        },
        {
          "name": "config",
          "value": "sfdc.json" // file in the archive
        }
      ]
    }
  ]
}

The UI also gets the spec signature to validate the spec:

Code Block
GET /4.0.0/packages/sfdc-lead-dump/1.0.1/spec.json.asc

The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /4.0.0/packages/sfdc-lead-dump/1.0.1/archive.tgz GET /4.0.0/packages/sfdc-lead-dump/1.0.1/archive.tgz.asc

...

When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:

Code Block

GET /4.0.0/packages.json
[
  ...,
  {
    "name": "mysql-jdbc-driver",
    "label": "MySQL JDBC Driver",
    "description": "JDBC Driver for MySQL databases.",
    "author": "MySQL",
    "org": "Oracle",
    "version": "5.1.39",
    "categories": [ "hydrator-plugins" ]
  },
  ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/license.txt [ gpl license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/spec.json
{    
  "name": "mysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
  "description": "JDBC Driver for MySQL databases.",
  "author": "MySQL",
  "org": "Oracle",
  "version": "5.1.39",
  "categories": [ "hydrator-plugins" ]
  "created": 1234567899,
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "name",
          "value": "mysql-connector-java"
        },
        {
          "name": "version",
          "value": "5.1.39"
        },
        {
          "name": "parents",
          "value": "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]"
        },
        {
          "name": "externalArchive"
          "value": "https://dev.mysql.com/downloads/file/?id=462849"
        },
        {
          "name": "externalArchiveSignature",
          "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz"
        },
        {
          "name": "externalArchiveJar",
          "value": "mysql-connector-java-5.1.39.zip.gz"
        },
        {
          "name": "config",
          "value": "mysql-connector-java-5.1.39.json" // file in the archive
        }
      ]
    }
  ]
}

The UI also makes a call to get the spec signature to make sure it is valid:

Code Block
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc

The UI then makes calls to get the archive and its signature to validate the archive, and unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.

Code Block
GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/archive.zip.asc GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/archive.zip

Versions Compared

Old Version 41

New Version 42

Key

Get Package Archive

Get Package Archive Signature

Get Package Archive Signature

Get Package Spec

Get Package Spec Signature

Get Package Icon

Get Package License

Package Spec

Dependencies

install_package

Failures

Hosting a Custom Marketplace

Page Comparison

Versions Compared

Old Version 41

New Version 42

Key

Get Package Archive

Get Package Archive Signature

Get Package Archive Signature

Get Package Spec

Get Package Spec Signature

Get Package Icon

Get Package License

Package Spec

Dependencies

install_package

Failures

Hosting a Custom Marketplace