...
| Code Block |
|---|
GET
<base>/<cdap-version>/categories.json
<base>/<cdap-version>/packages.json
<base>/<cdap-version>/packages/<package-name>/<version>/icon.jpg
<base>/<cdap-version>/packages/<package-name>/<version>/license.txt
<base>/<cdap-version>/packages/<package-name>/<version>/spec.json
<base>/<cdap-version>/packages/<package-name>/<version>/spec.json.asc
<base>/<cdap-version>/packages/<package-name>/<version>/archive.zip
<base>/<cdap-version>/packages/<package-name>/<version>/archive.zip.asc |
The packages.json and signature files could be generated from all the package spec.json files using a tool.
List
...
all Packages
| Code Block |
|---|
GET /<cdap-version>/categoriespackages.json ex: /4.0.0/categoriespackages.json [ { "name": "examplesPurchaseExample", "label": "ExamplesPurchase History", "description": "Example applicationsApplication todemonstrating getusage startedof with CDAP." }, {flows, workflows, mapreduce, and services.", "nameauthor": "use-casesCask", "labelorg": "Use CasesCask Data Inc.", "descriptionversion": "Common Use Cases."4.0.1", }, ... ] |
List all Packages
| Code Block |
|---|
GET /<cdap-version>/packages.json ex: /4.0.0/packages.json [ "categories": [ "examples" ] }, { "name": "PurchaseExampleHelloWorld", "label": "PurchaseHello HistoryWorld", "description": "ExampleSimple Applicationapplication demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.10", "categories": [ "examples" ] }, { "name": "HelloWorld", "label": "Hello World", "description": "Simple application demonstrating usage of flows and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.0", "categories": [ "examples" ] }, ... ] |
| Note |
|---|
This leaves grouping by category up to the UI. If needed, we could perhaps add packages-<category>.json files that only list the packages in a specific category. This also leaves display of multiple versions of the same package up to the UI. Though it seems like most of the time we would only have one version of the package per cdap version so maybe it's not a big problem. |
Get Package Archive
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/archive.zip
ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] |
Get Package Archive Signature
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/archive.zip.asc
ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ] |
Get Package Spec
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/spec.json ex: GET /4.0.0/packages/PurchaseExample/4.0.0/spec.json { "spec-version": "1.0", "name": "PurchaseExample", "label": "Purchase History", "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.0", "created": 1234567899, "changelog": "fixed a small parsing bug", "categories": [ "examples" ], "dependencies": { }, "actions": [ { ... ] |
| Note |
|---|
This leaves grouping by category up to the UI. If needed, we could perhaps add packages-<category>.json files that only list the packages in a specific category. This also leaves display of multiple versions of the same package up to the UI. Though it seems like most of the time we would only have one version of the package per cdap version so maybe it's not a big problem. |
Get Package Archive
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/archive.zip
ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] |
Get Package Archive Signature
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/archive.zip.asc
ex: GET /4.0.0/packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ] |
Get Package Spec
| Code Block |
|---|
GET /<cdap-version>/packages/<package-name>/<version>/spec.json
ex: GET /4.0.0/packages/PurchaseExample/4.0.0/spec.json
{
"spec-version": "1.0",
"name": "PurchaseExample",
"label": "Purchase History",
"description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "4.0.0",
"created": 1234567899,
"changelog": "fixed a small parsing bug",
"categories": [ "examples" ],
"dependencies": { },
"actions": [
{
"type": "create_artifact",
"arguments": [
{
"name": "name",
"value": "PurchaseHistoryExample"
},
{
"name": "version",
"value": "4.0.1"
},
{
"name": "scope",
"value": "user"
},
{
"name": "jar",
"value": "PurchaseHistoryExample-4.0.1.jar"
}
]
},
{
"type": "create_app",
"arguments": [
{
"name": "name",
"default": "PurchaseHistory"
}
]
}
]
} |
...
| Note |
|---|
Package dependencies introduces non-trivial logic in the UI and allow users to create some complex dependency chains. To simplify things in the first version, it may be a good idea to enforce that dependencies are only one level deep. That is, a package cannot depend on a package that has dependencies. |
Failures
Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.
Example Use Cases
Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline
When the user clicks on the '+' button, the UI makes a call:
| Code Block |
|---|
GET /groups
[
{
"name": "examples",
"label": "Examples",
"description": "Example applications to get started with CDAP."
},
{
"name": "hydrator-pipelines",
"label": "Hydrator Pipelines",
"description": "Templates of various Hydrator pipelines."
},
...
] |
...
dependencies. |
Failures
Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.
Example Use Cases
Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline
When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:
| Code Block |
|---|
GET /groups/hydrator-plugins/packages4.0.0/packages.json [ ..., { "name": "sfdc-lead-dump", "label": "SFDC Lead Dump", "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.", "author": "Cask", "org": "Cask Data Inc." }, ... ] |
to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Pipelines', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Pipelines' group:
Among that list is the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get all versions of that package:
...