Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 20 Next »

Terminology

group - A collection of packages. When somebody clicks on the '+' button, they will see the list of groups from which they can choose and add a package. For example, 'Hydrator Plugins', or 'Example Applications' could both be groups.

package - A collection of entities (artifacts, applications, datasets, streams) to add to CDAP. A package is identified by a name and version, and belongs to a group. A package consists of an archive of resources (tarball) and a package spec.

package spec - A json file containing a list of actions to perform against CDAP. For example, a spec for the Purchase History example will include an action to add the Purchase History artifact, then an action to create an application from that artifact.

package archive - A tarball containing any resources needed to perform the actions in the package spec. For example, if the spec contains an action to add an artifact, the archive must contain the jar file to add.

APIs

All APIs are relative to a base path. For example, cask.co/marketplace/v1. In the initial version, there will only be GET APIs. 

List Groups

GET /groups
ex: GET /groups
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydrator-plugins",
    "label": "Hydrator Plugins",
    "description": "Collections of plugins to extend Hydrator functionality."
  },
  ...
]

Get Group Icon

GET /groups/<group>/icon
ex: GET /groups/examples/icon 
[binary image contents]

List Packages in a Group

GET /groups/<group>/packages
ex: GET /groups/examples/packages
[
  {
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  {
    "name": "HelloWorld",
    "label": "Hello World",
    "description": "Simple application demonstrating usage of flows and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  ...
]

Get Package Icon

GET /groups/<group>/packages/<package>/icon
ex: GET /groups/examples/packages/PurchaseExample/icon
[binary image contents]

List Package Versions

GET /groups/<group>/packages/<package>/versions
ex: GET /groups/examples/packages/PurchaseExample/versions
[
  {    
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.1",
    "created": 1234567899,
    "changelog": [
      "fixed a small parsing bug"
    ]
  },
  {    
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc."
    "version": "4.0.0",
    "created": 1234567890,
    "changelog": [
      "updated APIs to work with CDAP 4.0.0"
    ]
  },
  ...
]

Get Package Archive

GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz
[ binary archive contents] 

Get Package Archive Signature

GET /groups/<group>/packages/<package>/versions/<version>/archive.tgz.asc
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/archive.tgz.asc
[ archive signature ]

Get Package Spec

GET /groups/<group>/packages/<package>/versions/<version>/spec
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/spec
{
  "metadata": {
    "spec-version": "1.0",
  },
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.1",
  "created": 1234567899,
  "changelog": [
    "fixed a small parsing bug"
  ],
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "widget-type": "constant",
          "name": "name",
          "value": "PurchaseHistoryExample"
        },
        {
          "widget-type": "constant",
          "name": "version",
          "value": "4.0.1"
        },
        {
          "widget-type": "constant",
          "name": "scope",
          "value": "user"
        },
        {
          "widget-type": "constant",
          "name": "jar",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "widget-type": "textbox",
          "name": "name",
          "default": "PurchaseHistory"
        }
      ]
    }
  ],
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    }
  }
}

Get Package Spec Signature

GET /groups/<group>/packages/<package>/versions/<version>/spec.asc
ex: GET /groups/examples/packages/PurchaseExample/versions/4.0.1/spec.asc
[ spec signature ]

Security

Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package.

Package Spec

The package spec contains some metadata about the spec itself, and a list of actions to perform on the CDAP instance. It is a JSON file of the following structure:

{
  "metadata": {
    "spec-version": "1.0"
  },
  "actions": [
    actionspec1,
    actionspec2,
    ...
  ]
}

The actions in the spec will correspond to steps in the UI wizard for installing the package.

Action Spec

Each action will contain a type, a list of arguments, and dependencies. Each type of action will require different arguments. In the first version, the following types will be supported: create_artifact, create_app, create_stream, create_dataset, create_hydrator_draft.

{
  "type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft",
  "arguments": [
    {
      "name": [argument name],
      "value": [argument value],
      "canModify": true | false
    }
  ]
}

Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.

create_artifact

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/artifact.html#add-an-artifact

namedescriptionrequired?default
nameartifact nameyes 
jarname of jar file in package archive

yes

 
archivelinklink to download 3rd party archivenonone
archivesiglink to get 3rd party archive signaturenonone
scopeartifact scope (implies API to add system artifacts is added in 4.0)nouser
versionartifact version to pass as Artifact-Version headernonone
parentsartifact parents to pass as Artifact-Extends headernonone
pluginsartifact plugins to pass as Artifact-Plugins headernonone

create_app

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#create-an-application

namedescriptionrequired?default
nameapp nameyes 
artifactscope, name, version of the artifact to create the app with

yes

 
configapp config (file in the package archive)noempty

create_stream

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#creating-a-stream

Depending on the arguments, subsequent calls to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#getting-and-setting-stream-properties (to set format, schema, ttl)

and http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#sending-events-to-a-stream-in-batch (load data into a stream) may be made.

namedescriptionrequired?default
namestream nameyes 
descriptionstream description, results in call to set stream propertiesnoempty
formatstream format as json object, results in call to set stream properties

no

empty
schemastream schema, results in call to set stream propertiesnoempty
ttlstream ttl, results in call to set stream propertiesno

empty

notification.threshold.mbmb threshold for sending notifications, results in call to set stream propertiesno

empty

loadfilesfiles in the package archive to write to the stream. results in a call to write to the stream in batchnoempty

create_dataset

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/dataset.html#creating-a-dataset

namedescriptionrequired?default
namedataset nameyes 
typedataset typeyes 
descriptiondataset descriptionnoempty
propertiesjson map of dataset properties

no

empty

create_hydrator_draft

Results in whatever the UI does to create a draft

namedescriptionrequired?default
namepipeline nameyes 
artifactscope, name, version of the artifact to create the app with

yes

 
configpipeline config (file in the package archive)yes 

Dependencies

Packages will only be able to specify dependencies on the CDAP version, as well as dependencies on the existence of specific CDAP entities. For example, the core-plugins-1.5.0 package requires that there exist system artifacts cdap-data-pipeline-4.0.0 and cdap-data-streams-4.0.0 in the CDAP instance.

{
  ...
  "dependencies": {
    "cdap": {
      "minVersion": "4.0.0",
      "maxVersion": "4.1.0"
    },
    "artifacts": [
      { 
        "scope": "system",
        "name": "spark-plugins",
        "minVersion": "1.5.0",
        "maxVersion": "1.6.0"
      },
      ...
    ],
    "streams": [
      { "name": "smsTexts" }
    ],
    "datasets": [
      { "name": "spamTexts" }
    ]
  }
}

Min versions are inclusive and max versions are exclusive.

Failures

Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.

Architecture

There will be a set of marketplace APIs that the UI will use to get groups, packages, package versions, icons, and package tarballs. There will be a market server that powers these APIs.  The server will use a set of internal storage interfaces that define how to read the information required by the APIs. We can start with a storage implementation that simply reads from local files, and perhaps another storage implementation that reads from cloud storage like S3.

The market server will be stateless, so a load balancer can be placed in front of it to ensure that it is highly available and to ensure that it can handle a high volume of requests

File Store

The first implementation of the storage layer can simply be a store that looks at a filesystem for files containing the relevant information. The File Store will expect a specific directory structure:

<base dir>/<group>/icon.jpg
<base dir>/<group>/meta.json
<base dir>/<group>/<package>/<version>/spec.json
<base dir>/<group>/<package>/<version>/icon.jpg
<base dir>/<group>/<package>/<version>/archive.tgz
 
ex:
/opt/cdap/marketplace/examples/icon.jpg
/opt/cdap/marketplace/examples/meta.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.1/icon.jpg
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/archive.tgz
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/spec.json
/opt/cdap/marketplace/examples/PurchaseExample/4.0.0/icon.jpg

On start up, the server will scan the base directory, load relevant information into memory, and simply serve data based on the contents of the files. This would also let ops teams manage the marketplace through use of 'group' packages and 'cask package' packages. 

 

Note: with such little logic in the server, why bother having a server at all? Why not just stick an Apache server in front of files or serve directly from S3? The assumption is that we will need to add more complicated functionality in the future, such as APIs to add groups and packages, ability to search for packages by various fields, etc.

Example Use Cases

Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline

When the user clicks on the '+' button, the UI makes a call:

GET /groups
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydrator-pipelines",
    "label": "Hydrator Pipelines",
    "description": "Templates of various Hydrator pipelines."
  },
  ...
]

to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Pipelines', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Pipelines' group:

GET /groups/hydrator-plugins/packages
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc."
  },
  ...
]

Among that list is the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get all versions of that package:

GET /groups/hydrator-plugins/packages/sfdc-lead-dump/versions
[
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.1",
    "created": 1234567899,
    "changelog": [
      "fixed a small parsing bug"
    ],
    "dependencies": {
      "cdap": {
        "minVersion": "4.0.0",
        "maxVersion": "4.1.0"
      }
    }
  },
  ...
]

It defaults to the most recent version that is compatible with the version of CDAP that is running. The user decides to install the package, so the UI makes a call to get the package spec:

GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec
{
  "name": "sfdc-lead-dump",
  "label": "SFDC Lead Dump",
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "1.0.1",
  "created": 1234567899,
  "changelog": [
    "fixed a small parsing bug"
  ],
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user",
          "canModify": false
        },
        {
          "name": "name",
          "value": "sfdc-plugins",
          "canModify": false
        },
        {
          "name": "version",
          "value": "1.0.0",
          "canModify": false
        },
        {
          "name": "parents",
          "value": "system:cdap-data-pipeline[4.0.0,4.1.0)",
          "canModify": false
        },
        {
          "name": "jar",
          "value": "sfdc-plugins.jar", // file in the archive
          "canModify": false
        }
      ]
    },
    {
      "type": "create_hydrator_draft",
      "arguments": [
        {
          "name": "artifact",
          "value": {
            "scope": "system",
            "name": "cdap-data-pipeline",
            "version": "4.0.0"
          },
          "canModify": false
        },
        {
          "name": "name",
          "value": "SFDC Lead Dump",
          "canModify": true
        },
        {
          "name": "config",
          "value": "sfdc.json", // file in the archive
          "canModify": false
        }
      ]
    }
  ]
}

The UI also fetches the spec signature and uses the public key to validate the spec:

GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec.asc

The UI also fetch the package archive and signature. It validates the package, and writes the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator draft.

 

Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.

When the user clicks on the '+' button, the UI makes a call:

GET /groups
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydrator-plugins",
    "label": "Hydrator Plugins",
    "description": "Plugins for Hydrator Pipelines."
  },
  ...
]

to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Plugins', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Plugins' group:

GET /groups/hydrator-plugins/packages
[
  ...,
  {
    "name": "mysql-jdbc-driver",
    "label": "MySQL JDBC Driver",
    "description": "JDBC Driver for MySQL databases.",
    "author": "MySQL",
    "org": "Oracle"
  },
  ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get all versions of that package:

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions
[
  {
    "name": "mysql-jdbc-driver",
    "label": "MySQL JDBC Driver",
    "description": "JDBC Driver for MySQL databases.",
    "author": "MySQL",
    "org": "Oracle",
    "version": "5.1.39",
    "created": 1234567899,
    "changelog": [ ],
    "dependencies": { }
  },
  ...
]

The user decides to install the 5.1.38 version of the driver. The UI makes a call to get the spec, and to get the spec signature to make sure it is valid:

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec
{    
  "name": "mysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
  "description": "JDBC Driver for MySQL databases.",
  "author": "MySQL",
  "org": "Oracle",
  "version": "5.1.39",
  "created": 1234567899,
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user",
          "canModify": false
        },
        {
          "name": "name",
          "value": "mysql-connector-java",
          "canModify": false
        },
        {
          "name": "version",
          "value": "5.1.39",
          "canModify": false
        },
        {
          "name": "parents",
          "value": "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]",
          "canModify": false
        },
        {
          "name": "jar",
          "value": "mysql-connector-java-5.1.39-bin.jar", // file in the archive
          "canModify": false
        },
        {
          "name": "archivelink"
          "value": "https://dev.mysql.com/downloads/file/?id=462849",
          "canModify": false
        },
        {
          "name": "archivesig",
          "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.tar.gz"
        }
        {
          "name": "plugins",
          "value": {
            "parents": [ "cdap-data-pipeline[3.0.0,10.0.0]" ],
            "plugins": [
              {
                "name" : "mysql",
                "type" : "jdbc",
                "className" : "com.mysql.jdbc.Driver",
                "description" : "Plugin for MySQL JDBC driver"
              }
            ]
          },
          "canModify": false
        }
      ]
    }
  ]
}

The UI then makes calls to get the archive and its signature to validate the archive, and unpack it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.38/archive.tgz.asc
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.38/archive.tgz
  • No labels