...

package spec - A json file containing a list of actions to perform against CDAP. For example, a spec for the Purchase History example will include an action to add the Purchase History artifact, then an action to create an application from that artifact.

package archive - A tarball zip containing any resources needed to perform the actions in the package spec. For example, if the spec contains an action to add an artifact, the archive must contain the jar file to add.

category - A package can be tagged with one or more categories. A category corresponds to one of the tabs on the left bar of the mocks.

Architecture

There will be a set of marketplace APIs that the UI will use to get categories and packages. In the first version of the market, the APIs will simply be static content served from S3. There will be an internal process to push the entire market repository to S3. If a user wishes to host their own marketplace, they can do so using their own S3 instance or by hosting their own server.

APIs

All APIs are relative to a base path. For example, cask.co/marketplace/v1. In the initial version, there will only be GET APIs. 

List Groups

List Categories

GET /groups
ex: GET /groups
categories.json
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydratoruse-pluginscases",
    "label": "HydratorUse PluginsCases",
    "description": "CollectionsCommon of plugins to extend Hydrator functionalityUse Cases."
  },
  ...
]

...

List all Packages

GET /groups/<group>/icon
ex: GET /groups/examples/icon 
[binary image contents]

List Packages in a Group

GET /groups/<group>/packages
ex: GET /groups/examples/packages
[
  {
    packages.json
[
  {
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  {
    "name": "HelloWorld",
    "label": "Hello World",
    "description": "Simple application demonstrating usage of flows and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  ...
]

...

List Packages in a Category

GET /groups/<group>/packages/<package>/iconpackages-<category>.json
ex: GET /groups/examples/packages/PurchaseExample/icon
[binary image contents]

List Package Versions

GET /groups/<group>/packages/<package>/versions
ex: GET /groups/examples/packages/PurchaseExample/versions
[
  {    
    packages-examples.json
[
  {
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "versioncurrentVersion": "4.0.10"
  },
  {
    "createdname": 1234567899"HelloWorld",
    "changeloglabel": ["Hello World",
     "fixed a small parsing bug"
    ]
  },
  {  "description": "Simple application demonstrating usage of flows and services.",
      "nameauthor": "PurchaseExampleCask",
    "labelorg": "Purchase HistoryCask Data Inc.",
    "descriptioncurrentVersion": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",4.0.0"
  },
  ...
]

List Package Versions

GET /packages/<package-name>/versions.json
ex: GET /packages/PurchaseExample/versions.json
[
  {    
    "authorname": "CaskPurchaseExample",
    "orglabel": "CaskPurchase Data History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.01",
    "created": 12345678901234567899,
    "changelog": [
      "updatedfixed APIsa tosmall work with CDAP 4.0.0parsing bug"
    ]
  },
  ...
]

Get Package Archive

GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz
[ binary archive contents] 

Get Package Archive Signature

GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz.asc
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz.asc
[ archive signature ] 

Get Package Spec

GET /groups/<group>/packages/<package>/versions/<version>/spec
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/spec
{
  "metadata": {
    "spec-version": "1.0",
  },
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.1",
  "created": 1234567899,
  "changelog": [
    "fixed a small parsing bug"
  ],
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "name",
          "value": "PurchaseHistoryExample"
        },
        {
          "name": "version",
   {    
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.0",
    "created": 1234567890,
    "changelog": [
      "updated APIs to work with CDAP 4.0.0"
    ]
  },
  ...
]

Get Package Archive

GET /packages/<package-name>/<version>/archive.zip
ex: GET /packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] 

Get Package Archive Signature

GET /packages/<package-name>/<version>/archive.zip.asc
ex: GET /packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ] 

Get Package Spec

GET /packages/<package>/<version>/spec.json
ex: GET /packages/PurchaseExample/4.0.1/spec.json
{
  "spec-version": "1.0",
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.1",
  "created": 1234567899,
  "changelog": [
    "fixed a small parsing bug"
  ],
  "actions": [
    {
      "valuetype": "4.0.1"create_artifact",
        },"arguments": [
        {
          "name": "scopename",
          "value": "userPurchaseHistoryExample"
        },
        {
          "name": "jarversion",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        },
      ]     },{
    {       "typename": "create_appscope",
      "arguments    "value": [ "user"
        },
        {
          "name": "namejar",
          "defaultvalue": "PurchaseHistoryPurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    ],{
      "dependenciestype": {"create_app",
      "cdaparguments": [
{         "minVersion": "4.0.0",{
          "maxVersionname": "4.1.0name",
    }   }
}

Get Package Spec Signature

GET /groups/<group>/packages/<package>/versions/<version>/spec.asc
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/spec.asc
[ spec signature ]

Security

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package.

Package Spec

The package spec contains some metadata about the spec itself, and a list of actions to perform on the CDAP instance. It is a JSON file of the following structure:

{
  "metadata": {
    "spec-version": "1.0"
  },
  "actions": [
    actionspec1,
    actionspec2,
    ...
  ]
}

The actions in the spec will correspond to steps in the UI wizard for installing the package.

Action Spec

...

   "default": "PurchaseHistory"
        }
      ]
    }
  ],
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    }
  }
}

Get Package Spec Signature

GET /packages/<package>/<version>/spec.asc
ex: GET /packages/PurchaseExample/4.0.1/spec.asc
[ spec signature ]

Security

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package.

Package Spec

The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:

{
  "typespec-version": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft"1.0"
  "actions": [
    actionspec1,
  "arguments": [
    {
      "name": [argument name] actionspec2,
    ...
 "value": [argument value],
      "canModify": true | false
    }
 
]
}

Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.

create_artifact

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/artifact.html#add-an-artifact

...

yes

...

The actions in the spec will correspond to steps in the UI wizard for installing the package.

Action Spec

Each action will contain a type, a list of arguments, and dependencies. Each type of action will require different arguments. In the first version, the following types will be supported: create_artifact, create_app, create_stream, create_dataset, create_hydrator_draft.

{
  "type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft",
  "arguments": [
    {
      "name": [argument name],
      "value": [argument value],
      "canModify": true | false
    }
  ]
}

Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.

create_artifact

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/artifact.html#add-an-artifact

...

create_hydrator_draft

Results in whatever the UI does to create a draft

Dependencies

Packages will only be able to specify dependencies on the CDAP version, as well as dependencies on the existence of specific CDAP entities. For example, the core-plugins-1.5.0 package requires that there exist system artifacts cdap-data-pipeline-4.0.0 and cdap-data-streams-4.0.0 in the CDAP instance.

{
  ...
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    },
    "artifacts": [
      { 
        "scope": "system",
        "name": "spark-plugins",
        "minVersion": "1.5.0",
        "maxVersion": "1.6.0"
      },
      ...
    ],
    "streams": [
      { "name": "smsTexts" }
    ],
    "datasets": [
      { "name": "spamTexts" }
    ]
  }
}

Min versions are inclusive and max versions are exclusive.

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.

Architecture

There will be a set of marketplace APIs that the UI will use to get groups, packages, package versions, icons, and package tarballs. There will be a market server that powers these APIs.  The server will use a set of internal storage interfaces that define how to read the information required by the APIs. We can start with a storage implementation that simply reads from local files, and perhaps another storage implementation that reads from cloud storage like S3.

Image Removed

The market server will be stateless, so a load balancer can be placed in front of it to ensure that it is highly available and to ensure that it can handle a high volume of requests

Image Removed

File Store

The first implementation of the storage layer can simply be a store that looks at a filesystem for files containing the relevant information. The File Store will expect a specific directory structure:

<base dir>/<group>/icon.jpg
<base dir>/<group>/meta.json
<base dir>/<group>/<package>/<version>/spec.json
<base dir>/<group>/<package>/<version>/icon.jpg
<base dir>/<group>/<package>/<version>/archive.tgz
 
ex:
/opt/cdap/marketplace/examples/icon.jpg
/opt/cdap/marketplace/examples/meta.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/icon.jpg
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/icon.jpg

On start up, the server will scan the base directory, load relevant information into memory, and simply serve data based on the contents of the files. This would also let ops teams manage the marketplace through use of 'group' packages and 'cask package' packages. 

...

create_hydrator_draft

Results in whatever the UI does to create a draft

Dependencies

Packages will only be able to specify dependencies on the CDAP version, as well as dependencies on the existence of specific CDAP entities. For example, the core-plugins-1.5.0 package requires that there exist system artifacts cdap-data-pipeline-4.0.0 and cdap-data-streams-4.0.0 in the CDAP instance.

{
  ...
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    },
    "artifacts": [
      { 
        "scope": "system",
        "name": "spark-plugins",
        "minVersion": "1.5.0",
        "maxVersion": "1.6.0"
      },
      ...
    ],
    "streams": [
      { "name": "smsTexts" }
    ],
    "datasets": [
      { "name": "spamTexts" }
    ]
  }
}

Min versions are inclusive and max versions are exclusive.

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.

Example Use Cases

Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline

...