...
The APIs are simply a contract about the directory structure of the marketplace. All APIs are relative to a base path. For example, cask.co/marketplace/v1. If we ever need to change the contract, we can change the base to a higher version. The structure is expected to be:
| Code Block |
|---|
GET
<base>/v1/packages.json
<base>/v1/packages/<package-name>/<version>/icon.jpg
<base>/v1/packages/<package-name>/<version>/license.txt
<base>/v1/packages/<package-name>/<version>/spec.json
<base>/v1/packages/<package-name>/<version>/spec.json.asc
<base>/v1/packages/<package-name>/<version>/archive.zip
<base>/v1/packages/<package-name>/<version>/archive.zip.asc |
...
List all Packages
| Code Block |
|---|
GET /v1/<cdap-version>/packages.json
ex: GET /v1/packages.json
[
{
"name": "PurchaseExample",
"label": "Purchase History",
"description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "4.0.1",
"categories": [ "examples" ],
"cdapVersion": "[4.0.0,4.1.0)
},
{
"name": "HelloWorld",
"label": "Hello World",
"description": "Simple application demonstrating usage of flows and services.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "4.0.0",
"categories": [ "examples" ],
"cdapVersion": "[4.0.0,4.1.0)"
},
...
] |
...
Get Package Archive
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/archive.zip
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] |
Get Package Archive Signature
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/archive.zip.asc
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ] |
Get Package Spec
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/spec.json
ex: GET /v1/packages/PurchaseExample/4.0.0/spec.json
{
"specVersion": "1.0",
"name": "PurchaseExample",
"label": "Purchase History",
"description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "4.0.0",
"created": 1234567899,
"cdapVersion": "[4.0.0,4.1.0)",
"changelog": "fixed a small parsing bug",
"categories": [ "examples" ],
"actions": [
{
"type": "create_artifact",
"arguments": [
{
"name": "name",
"value": "PurchaseHistoryExample"
},
{
"name": "version",
"value": "4.0.1"
},
{
"name": "scope",
"value": "user"
},
{
"name": "jar",
"value": "PurchaseHistoryExample-4.0.1.jar"
}
]
},
{
"type": "create_app",
"arguments": [
{
"name": "name",
"default": "PurchaseHistory"
}
]
}
]
} |
Get Package Spec Signature
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/spec.asc
ex: GET /v1/packages/PurchaseExample/4.0.0/spec.asc
[ spec signature ] |
Get Package Icon
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/icon.jpg
ex: GET /v1/packages/PurchaseExample/4.0.0/icon.jpg
[ icon bytes ] |
Get Package License
| Code Block |
|---|
GET /v1/packages/<package-name>/<version>/license.txt
ex: GET /v1/packages/PurchaseExample/4.0.0/license.txt
Copyright © 2014-2016 Cask Data, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
... |
...
When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:
| Code Block |
|---|
GET /v1/packages.json
[
...,
{
"name": "sfdc-lead-dump",
"label": "SFDC Lead Dump",
"description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "1.0.0",
"categories": [ "hydrator-pipelines" ]
},
...
] |
Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get the license for that package:
| Code Block |
|---|
GET /v1/packages/sfdc-lead-dump/1.0.0/license.txt
[ apache2 license ] |
...
The user accepts the conditions, and the UI makes a call to get the spec for that package:
| Code Block |
|---|
GET /v1/packages/sfdc-lead-dump/1.0.0/spec.json
{
"name": "sfdc-lead-dump",
"label": "SFDC Lead Dump",
"description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "1.0.1",
"created": 1234567899,
"changelog": "",
"actions": [
{
"type": "create_artifact",
"arguments": [
{
"name": "scope",
"value": "user"
},
{
"name": "name",
"value": "sfdc-plugins"
},
{
"name": "version",
"value": "1.0.0"
},
{
"name": "config",
"value": "sfdc-plugins.json" // file in the archive
},
{
"name": "jar",
"value": "sfdc-plugins.jar" // file in the archive
}
]
},
{
"type": "create_app",
"arguments": [
{
"name": "artifact",
"value": {
"scope": "system",
"name": "cdap-data-pipeline",
"version": "4.0.0"
}
},
{
"name": "name",
"value": "SFDC Lead Dump",
"canModify": true
},
{
"name": "config",
"value": "sfdc.json" // file in the archive
}
]
}
]
} |
The UI also gets the spec signature to validate the spec:
| Code Block |
|---|
GET /v1/packages/sfdc-lead-dump/1.0.1/spec.json.asc |
The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft
| Code Block |
|---|
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip.asc |
...
When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:
| Code Block |
|---|
GET /v1/packages.json
[
...,
{
"name": "mysql-jdbc-driver",
"label": "MySQL JDBC Driver",
"description": "JDBC Driver for MySQL databases.",
"author": "MySQL",
"org": "Oracle",
"version": "5.1.39",
"categories": [ "hydrator-plugins" ]
},
...
] |
Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:
| Code Block |
|---|
GET /v1/packages/mysql-jdbc-driver/5.1.39/license.txt
[ gpl license ] |
The user accepts the conditions, and the UI makes a call to get the spec for that package:
| Code Block |
|---|
GET /v1/packages/mysql-jdbc-driver/5.1.39/spec.json
{
"name": "mysql-jdbc-driver",
"label": "MySQL JDBC Driver",
"description": "JDBC Driver for MySQL databases.",
"author": "MySQL",
"org": "Oracle",
"version": "5.1.39",
"categories": [ "hydrator-plugins" ]
"created": 1234567899,
"actions": [
{
"type": "create_artifact",
"arguments": [
{
"name": "scope",
"value": "user"
},
{
"name": "name",
"value": "mysql-connector-java"
},
{
"name": "version",
"value": "5.1.39"
},
{
"name": "externalArchive"
"value": "https://dev.mysql.com/downloads/file/?id=462849"
},
{
"name": "externalArchiveSignature",
"value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz"
},
{
"name": "externalArchiveJar",
"value": "mysql-connector-java-5.1.39-bin.jar"
},
{
"name": "config",
"value": "mysql-connector-java-5.1.39.json" // file in the archive containing parents and plugins
}
]
}
]
} |
The UI also makes a call to get the spec signature to make sure it is valid:
| Code Block |
|---|
GET /v1/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc |
The UI then makes calls to get the archive and its signature to validate the archive, and unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.
| Code Block |
|---|
GET /v1/packages/mysql-jdbc-driver/5.1.39/archive.zip.asc
GET /v1/packages/mysql-jdbc-driver/5.1.39/archive.zip |