Versions Compared

Old Version 43

changes.mady.by.user Albert Shau

Saved on

compared with

New Version Current

changes.mady.by.user Albert Shau

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The APIs are simply a contract about the directory structure of the marketplace. All APIs are relative to a base path. For example, cask.co/marketplace/v1. If we ever need to change the contract, we can change the base to a higher version. The structure is expected to be:

Code Block
GET
<base>/v1/packages.json
<base>/v1/packages/<package-name>/<version>/icon.jpgpng
<base>/v1/packages/<package-name>/<version>/license.txt
<base>/v1/packages/<package-name>/<version>/spec.json
<base>/v1/packages/<package-name>/<version>/spec.json.asc
<base>/v1/packages/<package-name>/<version>/archive.zip
<base>/v1/packages/<package-name>/<version>/archive.zip.asc

...

List all Packages

Code Block
GET /v1/<cdap-version>/packages.json
ex: GET /v1/packages.json
[
  {
    "name": "PurchaseExample",
    "label": "Purchase History",
    "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "4.0.1",
    "categories": [ "examples" ],
    "cdapVersion": "[4.0.0,4.1.0)
  },
  {
    "name": "HelloWorld",
    "label": "Hello World",
    "description": "Simple application demonstrating usage of flows and services.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "4.0.0",
    "categories": [ "examples" ],
    "cdapVersion": "[4.0.0,4.1.0)"
  },
  ...
]

...

Get Package Archive

Code Block
GET /v1/packages/<package-name>/<version>/archive.zip
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] 

Get Package Archive Signature

Code Block
GET /v1/packages/<package-name>/<version>/archive.zip.asc
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ]

Get Package Spec

Code Block
GET /v1/packages/<package-name>/<version>/spec.json
ex: GET /v1/packages/PurchaseExample/4.0.0/spec.json
{
  "spec-versionspecVersion": "1.0",
  "name": "PurchaseExample",
  "label": "Purchase History",
  "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "4.0.0",
  "created": 1234567899,
  "cdapVersion": "[4.0.0,4.1.0)",
  "changelog": "fixed a small parsing bug",
  "categories": [ "examples" ],
  "dependencies": { },
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "name",
          "value": "PurchaseHistoryExample"
        },
        {
          "name": "version",
          "value": "4.0.1"
        },
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "jar",
          "value": "PurchaseHistoryExample-4.0.1.jar"
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "name",
          "default": "PurchaseHistory"
        }
      ]
    }
  ]
}

Get Package Spec Signature

Code Block
GET /v1/packages/<package-name>/<version>/spec.asc
ex: GET /v1/packages/PurchaseExample/4.0.0/spec.asc
[ spec signature ]

Get Package Icon

Code Block
GET /v1/packages/<package-name>/<version>/icon.jpgpng
ex: GET /v1/packages/PurchaseExample/4.0.0/icon.jpgpng
[ icon bytes ]

Get Package License

Code Block
GET /v1/packages/<package-name>/<version>/license.txt
ex: GET /v1/packages/PurchaseExample/4.0.0/license.txt
Copyright © 2014-2016 Cask Data, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
       http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
...

...

The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:

Code Block
{
  "spec-versionspecVersion": "1.0"
  "actionsname": ["<name>",
    actionspec1"version": "<version>",
    actionspec2"label": "<label>",
  "description":  ..."<description>",
   ]
}"org": "<org>",
  "categories": [ <categories> ],
  "cdapVersion": "<compatible-versions>",
  "changelog": "<changes>",
  "actions": [
    actionspec1,
    actionspec2,
    ...
  ]
}

The actions in the spec will correspond to steps in the UI wizard for installing the package.

...

Code Block
{
  "type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "load_datapack" | "install_package"
  "arguments": [
    {
      "name": [argument name],
      "value": [argument value],
      "canModify": true | false (defaults to false)
    }
  ]
}

...

namedescriptionrequired?default
nameartifact nameyes 
jarname of jar file in package archive

no (if using externalArchive)

 
externalJarlink to download 3rd party jarnonone
externalArchivelink to download 3rd party archivenonone
externalArchiveSignaturelink to get 3rd party archive signaturenonone
externalArchiveJarpath of the jar file in the external archivenonone
scopeartifact scope (implies API to add system artifacts is added in 4.0)nouser
versionartifact version to pass as Artifact-Version headernonone
configconfig file contains artifact parents, plugins, and propertiesnonone

...

namedescriptionrequired?default
namestream nameyes 
descriptionstream description, results in call to set stream propertiesnoempty
formatstream format as json object, results in call to set stream properties

no

empty
schemastream schema, results in call to set stream propertiesnoempty
ttlstream ttl, results in call to set stream propertiesno

empty

notification.threshold.mbmb threshold for sending notifications, results in call to set stream propertiesno

empty

loadfilesfiles in the package archive to write to the stream. results in a call to write to the stream in batch.mbmb threshold for sending notifications, results in call to set stream propertiesno

empty

create_dataset

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/dataset.html#creating-a-dataset

namedescriptionrequired?default
namedataset nameyes 
typedataset typeyes 
descriptiondataset descriptionnoempty
propertiesjson map of dataset properties

no

empty

load_datapack

Loads a datapack into some dataset or stream.

namedescriptionrequired?default
namedataset/stream nameyes 
filesfiles to load into the dataset/streamyes 

install_package

Installs another package from the marketplace. May not be implemented for version 1.

namedescriptionrequired?default
namepackage nameyes 
versionpackage versionyes 

...

When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:

Code Block
GET /v1/packages.json
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.0",
    "categories": [ "hydrator-pipelines" ]
  },
  ...
]

Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.0/license.txt
[ apache2 license ]

...

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.0/spec.json
{
  "name": "sfdc-lead-dump",
  "label": "SFDC Lead Dump",
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask",
  "org": "Cask Data Inc.",
  "version": "1.0.1",
  "created": 1234567899,
  "changelog": "",
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "name",
          "value": "sfdc-plugins"
        },
        {
          "name": "version",
          "value": "1.0.0"
        },
        {
          "name": "config",
          "value": "sfdc-plugins.json" // file in the archive
        },
        {
          "name": "jar",
          "value": "sfdc-plugins.jar" // file in the archive
        }
      ]
    },
    {
      "type": "create_app",
      "arguments": [
        {
          "name": "artifact",
          "value": {
            "scope": "system",
            "name": "cdap-data-pipeline",
            "version": "4.0.0"
          }
        },
        {
          "name": "name",
          "value": "SFDC Lead Dump",
          "canModify": true
        },
        {
          "name": "config",
          "value": "sfdc.json" // file in the archive
        }
      ]
    }
  ]
}

The UI also gets the spec signature to validate the spec:

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.1/spec.json.asc

The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.tgzzip
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.tgzzip.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator pipeline.

...

When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:

Code Block
GET /v1/packages.json
[
  ...,
  {
    "name": "mysql-jdbc-driver",
    "label": "MySQL JDBC Driver",
    "description": "JDBC Driver for MySQL databases.",
    "author": "MySQL",
    "org": "Oracle",
    "version": "5.1.39",
    "categories": [ "hydrator-plugins" ]
  },
  ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /v1/packages/mysql-jdbc-driver/5.1.39/license.txt
[ gpl license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block
GET /v1/packages/mysql-jdbc-driver/5.1.39/spec.json
{    
  "name": "mysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
  "description": "JDBC Driver for MySQL databases.",
  "author": "MySQL",
  "org": "Oracle",
  "version": "5.1.39",
  "categories": [ "hydrator-plugins" ]
  "created": 1234567899,
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
        {
          "name": "scope",
          "value": "user"
        },
        {
          "name": "name",
          "value": "mysql-connector-javauser"
        },
        {
          "name": "versionname",
          "value": "5.1.39mysql-connector-java"
        },
        {
          "name": "parentsversion",
          "value": "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]5.1.39"
        },
        {
          "name": "externalArchive"
          "value": "https://dev.mysql.com/downloads/file/?id=462849"
        },
        {
          "name": "externalArchiveSignature",
          "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz"
        },
        {
          "name": "externalArchiveJar",
          "value": "mysql-connector-java-5.1.39-bin.zip.gzjar"
        },
        {
          "name": "config",
          "value": "mysql-connector-java-5.1.39.json" // file in the archive containing parents and plugins
        }
      ]
    }
  ]
}

The UI also makes a call to get the spec signature to make sure it is valid:

Code Block
GET /v1/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc

The UI then makes calls to get the archive and its signature to validate the archive, and unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.

Code Block
GET /v1/packages/mysql-jdbc-driver/5.1.39/archive.zip.asc
GET /v1/packages/mysql-jdbc-driver/5.1.39/archive.zip