...
There will be a set of marketplace APIs that the UI will use to get categories and packages. In the first version of the market, the APIs will simply be static content served from S3. This essentially amounts to placing packages in a pre-determined directory structure, and generating index files that will be used to get metadata about the packages, such as the list of all packages, list of packages in a category, versions of a package, etc.a file that lists all packages in the repository.
There will be an internal process to push the entire market repository to S3. If a user wishes to host their own marketplace, they can do so using their own S3 instance or by hosting their own serverby sticking a server (apache httpd for example) on top of a local directory structure.
APIs
The APIs are simply a contract about the directory structure of the marketplace. All APIs are relative to a base path. For example, cask.co/marketplace/v1.
Note |
---|
This will become increasingly difficult to manage as CDAP evolves. We may soon need a service to search for packages, filtering on categories or filtering out packages that are incompatible with a specific version of cdap, etc. |
List Categories
...
The structure is expected to be:
Code Block |
---|
GET
<base>/v1/packages.json
<base>/v1/packages/<package-name>/<version>/icon.png
<base>/v1/packages/<package-name>/<version>/license.txt
<base>/v1/packages/<package-name>/<version>/spec.json
<base>/v1/packages/<package-name>/<version>/spec.json.asc
<base>/v1/packages/<package-name>/<version>/archive.zip
<base>/v1/packages/<package-name>/<version>/archive.zip.asc |
The packages.json and signature files could be generated from all the package spec.json files using a tool.
List all Packages
Code Block |
---|
GET /v1/<cdap-version>/packages.json ex: GET /v1/packages.json [ { "name": "use-casesPurchaseExample", "label": "UsePurchase CasesHistory", "description": "Common Use Cases." }, ... ] |
List Latest Version of all Packages
Code Block |
---|
GET /packages.json [ { "name": "PurchaseExample", "label": "Purchase History", "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.1", "categories": [ "examples" ], "cdapVersion": "[4.0.0,4.1.0) }, { "name": "HelloWorld", "label": "Hello World", "description": "Simple application demonstrating usage of flows and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.0", "categories": [ "examples" ], }, "cdapVersion": "[4.0.. ] |
List Latest Version of all Packages in a Category
Code Block |
---|
GET /packages-<category>.json ex: GET /packages-examples.json [ { "name": "PurchaseExample", "label": "Purchase History", "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.1", "categories": [ "examples" ] }, { "name": "HelloWorld", "label": "Hello World", 0,4.1.0)" }, ... ] |
This list is not expected to change often. It can be cached by the UI if needed. The 'cdapVersion' specifies which versions of cdap the package is compatible with. If none is given, it is compatible with all versions.
Note |
---|
This leaves grouping by category up to the UI. If needed, we could perhaps add packages-<category>.json files that only list the packages in a specific category. This also leaves display of multiple versions of the same package up to the UI. Though it seems like most of the time we would only have one version of the package per cdap version so maybe it's not a big problem. This also leaves filtering of packages incompatible with the cdap instance up to the UI. |
Get Package Archive
Code Block |
---|
GET /v1/packages/<package-name>/<version>/archive.zip
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip
[ binary archive contents] |
Get Package Archive Signature
Code Block |
---|
GET /v1/packages/<package-name>/<version>/archive.zip.asc
ex: GET /v1/packages/PurchaseExample/4.0.1/archive.zip.asc
[ archive signature ] |
Get Package Spec
Code Block |
---|
GET /v1/packages/<package-name>/<version>/spec.json ex: GET /v1/packages/PurchaseExample/4.0.0/spec.json { "specVersion": "1.0", "name": "PurchaseExample", "label": "Purchase History", "description": "SimpleExample applicationApplication demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.0", "categoriescreated": [1234567899, "examplescdapVersion" ] }, ... ] |
List Package Versions
Code Block |
---|
GET /packages/<package-name>/versions.json ex: GET /packages/PurchaseExample/versions.json [ { "name": "PurchaseExample", "label": "Purchase History", "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask",: "[4.0.0,4.1.0)", "changelog": "fixed a small parsing bug", "categories": [ "examples" ], "actions": [ { "type": "create_artifact", "arguments": [ { "orgname": "Cask Data Inc.name", "version": "4.0.1", "categoriesvalue": [ "examplesPurchaseHistoryExample" ], "created": 1234567899, "changelog": "fixed a small parsing bug", "dependencies": }, { "cdap": { "minVersionname": "4.0.0version", "maxVersionvalue": "4.0.1.0" }, } }, { { "name": "PurchaseExamplescope", "label": "Purchase History", "descriptionvalue": "user"Example Application demonstrating usage of flows, workflows, mapreduce, and services."}, "author": "Cask", { "org": "Cask Data Inc.", "categoriesname": [ "examplesjar" ],, "versionvalue": "PurchaseHistoryExample-4.0.0",1.jar" "created": 1234567890, } "changelog": "updated APIs] to work with CDAP 4.0.0" }, "dependencies{ "type": {"create_app", "cdaparguments": [ { "minVersionname": "4.0.0name", "maxVersiondefault": "4.1.0PurchaseHistory" } } ] }, ...] ]} |
Get Package
...
Spec Signature
Code Block |
---|
GET /v1/packages/<package-name>/<version>/archivespec.zipasc ex: GET /v1/packages/PurchaseExample/4.0.10/archivespec.zipasc [ binaryspec archivesignature contents] |
Get Package
...
Icon
Code Block |
---|
GET /v1/packages/<package-name>/<version>/archiveicon.zip.ascpng ex: GET /v1/packages/PurchaseExample/4.0.10/archiveicon.zip.ascpng [ archiveicon signaturebytes ] |
Get Package
...
License
Code Block |
---|
GET /v1/packages/<package><package-name>/<version>/speclicense.jsontxt ex: GET /v1/packages/PurchaseExample/4.0.10/speclicense.jsontxt {Copyright © "spec-version": "1.0", "name": "PurchaseExample", "label": "Purchase History", "description": "Example Application demonstrating usage of flows, workflows, mapreduce, and services.", "author": "Cask", "org": "Cask Data Inc.", "version": "4.0.1", "created": 1234567899, "changelog": "fixed a small parsing bug", "categories": [ "examples" ], "dependencies": { "cdap": { "minVersion": "4.0.0", "maxVersion": "4.1.0" } }, "actions": [ { "type": "create_artifact", "arguments": [ { "name": "name", "value": "PurchaseHistoryExample" }, { "name": "version", "value": "4.0.1" }, { "name": "scope", "value": "user" }, { "name": "jar", "value": "PurchaseHistoryExample-4.0.1.jar" } ] }, { "type": "create_app", "arguments": [ { "name": "name", "default": "PurchaseHistory" } ] } ] } |
Get Package Spec Signature
Code Block |
---|
GET /packages/<package>/<version>/spec.asc
ex: GET /packages/PurchaseExample/4.0.1/spec.asc
[ spec signature ] |
Security
Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package.
Package Spec
The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:
Code Block |
---|
{
"spec-version": "1.0"
"actions": [
actionspec1,
actionspec2,
...
]
} |
The actions in the spec will correspond to steps in the UI wizard for installing the package.
Action Spec
Each action will contain a type, a list of arguments, and dependencies. Each type of action will require different arguments. In the first version, the following types will be supported: create_artifact, create_app, create_stream, create_dataset, create_hydrator_draft.
Code Block |
---|
{
"type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft",
"arguments": [
{
"name": [argument name],
"value": [argument value],
"canModify": true | false
}
]
} |
Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.
create_artifact
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/artifact.html#add-an-artifact
name | description | required? | default |
---|---|---|---|
name | artifact name | yes | |
jar | name of jar file in package archive | yes | |
archivelink | link to download 3rd party archive | no | none |
archivesig | link to get 3rd party archive signature | no | none |
scope | artifact scope (implies API to add system artifacts is added in 4.0) | no | user |
version | artifact version to pass as Artifact-Version header | no | none |
parents | artifact parents to pass as Artifact-Extends header | no | none |
plugins | artifact plugins to pass as Artifact-Plugins header | no | none |
create_app
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#create-an-application
name | description | required? | default |
---|---|---|---|
name | app name | yes | |
artifact | scope, name, version of the artifact to create the app with | yes | |
config | app config (file in the package archive) | no | empty |
create_stream
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#creating-a-stream
Depending on the arguments, subsequent calls to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#getting-and-setting-stream-properties (to set format, schema, ttl)
and http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#sending-events-to-a-stream-in-batch (load data into a stream) may be made.
name | description | required? | default |
---|---|---|---|
name | stream name | yes | |
description | stream description, results in call to set stream properties | no | empty |
format | stream format as json object, results in call to set stream properties | no | empty |
schema | stream schema, results in call to set stream properties | no | empty |
ttl | stream ttl, results in call to set stream properties | no | empty |
notification.threshold.mb | mb threshold for sending notifications, results in call to set stream properties | no | empty |
loadfiles | files in the package archive to write to the stream. results in a call to write to the stream in batch | no | empty |
create_dataset
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/dataset.html#creating-a-dataset
name | description | required? | default |
---|---|---|---|
name | dataset name | yes | |
type | dataset type | yes | |
description | dataset description | no | empty |
properties | json map of dataset properties | no | empty |
create_hydrator_draft
Results in whatever the UI does to create a draft
name | description | required? | default |
---|---|---|---|
name | pipeline name | yes | |
artifact | scope, name, version of the artifact to create the app with | yes | |
config | pipeline config (file in the package archive) | yes |
Dependencies
Packages will be able to specify dependencies on the CDAP version, as well as dependencies on other packages.
Code Block |
---|
{
...
"dependencies": {
"cdap": {
"minVersion": "4.0.0",
"maxVersion": "4.1.0"
},
"packages": [
{
"name": "spark-plugins",
"minVersion": "1.5.0",
"maxVersion": "1.6.0"
},
...
]
}
} |
Min versions are inclusive and max versions are exclusive.
If other packages are listed as a dependency, the actions for the dependent package must be executed before the actions of the current package.
Note |
---|
Package dependencies introduces non-trivial logic in the UI and allow users to create some complex dependency chains. To simplify things in the first version, it may be a good idea to enforce that dependencies are only one level deep. That is, a package cannot depend on a package that has dependencies. |
Failures
Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? Since the CDAP APIs backing these actions are idempotent, we can ask the user if they want to retry.
Example Use Cases
Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline
When the user clicks on the '+' button, the UI makes a call:
Code Block |
---|
GET /groups
[
{
"name": "examples",
"label": "Examples",
"description": "Example applications to get started with CDAP."
},
{
"name": "hydrator-pipelines",
"label": "Hydrator Pipelines",
"description": "Templates of various Hydrator pipelines."
},
...
] |
to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Pipelines', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Pipelines' group:
Code Block |
---|
GET /groups/hydrator-plugins/packages
[
...,
{
"name": "sfdc-lead-dump",
"label": "SFDC Lead Dump",
"description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
"author": "Cask",
"org": "Cask Data Inc."
},
...
] |
Among that list is the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get all versions of that package:
Code Block |
---|
GET /groups/hydrator-plugins/packages/sfdc-lead-dump/versions
[
{
"name": "sfdc-lead-dump",
"label": "SFDC Lead Dump",
"description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
"author": "Cask",
"org": "Cask Data Inc.",
"version": "1.0.1",
"created": 1234567899,
"changelog": [
"fixed a small parsing bug"
],
"dependencies": {
"cdap": {
"minVersion": "4.0.0",
"maxVersion": "4.1.0"
}
}
},
...
] |
It defaults to the most recent version that is compatible with the version of CDAP that is running. The user decides to install the package, so the UI makes a call to get the package spec:
Code Block |
---|
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec { "name": "sfdc-lead-dump", "label": "SFDC Lead Dump", "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.", "author": "Cask", "org": "Cask Data Inc.", "version": "1.0.1", "created": 1234567899, "changelog": [ "fixed a small parsing bug" ], "actions": [ { "type": "create_artifact", "arguments": [ { "name": "scope", "value": "user", "canModify": false }, { "name": "name", "value": "sfdc-plugins", "canModify": false }, { "name": "version", "value": "1.0.0", "canModify": false }, { "name": "parents", "value": "system:cdap-data-pipeline[4.0.0,4.1.0)", "canModify": false }, { "name": "jar", "value": "sfdc-plugins.jar", // file in the archive "canModify": false } ] }, { "type": "create_hydrator_draft", "arguments": [ { "name": "artifact", "value": { "scope": "system", "name": "cdap-data-pipeline", "version": "4.0.0" }, "canModify": false }, { "name": "name", "value2014-2016 Cask Data, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ... |
Security
Since people will be able to download code from the marketplace, it is especially important that there is protection against malicious code. We can make use of PGP in order to sign both the package archive and the package spec that are downloadable from the marketplace. The Market UI will have to be configured to use a GPG key (for the public CDAP marketplace, we could re-use the GPG key used for CDAP rpms and debians or create another one). It can then use that public key along with the signature APIs to verify that the spec and archive were signed by the owner of the package. There will also be a setting that lets people turn off signature checking in case its not needed for internally hosted repositories.
Package Spec
The package spec contains some metadata about the spec itself, and a list of steps to perform on the CDAP instance. It is a JSON file of the following structure:
Code Block |
---|
{
"specVersion": "1.0"
"name": "<name>",
"version": "<version>",
"label": "<label>",
"description": "<description>",
"org": "<org>",
"categories": [ <categories> ],
"cdapVersion": "<compatible-versions>",
"changelog": "<changes>",
"actions": [
actionspec1,
actionspec2,
...
]
} |
The actions in the spec will correspond to steps in the UI wizard for installing the package.
Action Spec
Each action will contain a type, a list of arguments, and dependencies. Each type of action will require different arguments. In the first version, the following types will be supported: create_artifact, create_app, create_stream, create_dataset.
Code Block |
---|
{
"type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "load_datapack" | "install_package"
"arguments": [
{
"name": [argument name],
"value": [argument value],
"canModify": true | false (defaults to false)
}
]
} |
Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.
create_artifact
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/artifact.html#add-an-artifact
name | description | required? | default |
---|---|---|---|
name | artifact name | yes | |
jar | name of jar file in package archive | no (if using externalArchive) | |
externalJar | link to download 3rd party jar | no | none |
externalArchive | link to download 3rd party archive | no | none |
externalArchiveSignature | link to get 3rd party archive signature | no | none |
externalArchiveJar | path of the jar file in the external archive | no | none |
scope | artifact scope (implies API to add system artifacts is added in 4.0) | no | user |
version | artifact version to pass as Artifact-Version header | no | none |
config | config file contains artifact parents, plugins, and properties | no | none |
create_app
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#create-an-application
name | description | required? | default |
---|---|---|---|
name | app name | yes | |
artifact | scope, name, version of the artifact to create the app with | yes | |
config | app config (file in the package archive) | no | empty |
create_stream
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#creating-a-stream
Depending on the arguments, subsequent calls to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#getting-and-setting-stream-properties (to set format, schema, ttl)
and http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/stream.html#sending-events-to-a-stream-in-batch (load data into a stream) may be made.
name | description | required? | default |
---|---|---|---|
name | stream name | yes | |
description | stream description, results in call to set stream properties | no | empty |
format | stream format as json object, results in call to set stream properties | no | empty |
schema | stream schema, results in call to set stream properties | no | empty |
ttl | stream ttl, results in call to set stream properties | no | empty |
notification.threshold.mb | mb threshold for sending notifications, results in call to set stream properties | no | empty |
create_dataset
Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/dataset.html#creating-a-dataset
name | description | required? | default |
---|---|---|---|
name | dataset name | yes | |
type | dataset type | yes | |
description | dataset description | no | empty |
properties | json map of dataset properties | no | empty |
load_datapack
Loads a datapack into some dataset or stream.
name | description | required? | default |
---|---|---|---|
name | dataset/stream name | yes | |
files | files to load into the dataset/stream | yes |
install_package
Installs another package from the marketplace.
name | description | required? | default |
---|---|---|---|
name | package name | yes | |
version | package version | yes |
Failures
Since a package spec can contain multiple actions, what happens if some actions succeed and then one action fails? We will not attempt rollback or anything like that. Instead, all the wizards that execute the actions must be idempotent. For example, if told to add an artifact and the artifact already exists, the step can simply be skipped.
Hosting a Custom Marketplace
To host a custom marketplace, users can run an apache httpd server on top of a local directory structure. To make this easier, we could create a github repository of all the public packages hosted by Cask. The repository will follow the directory structure documented here, and have a script at the top level that builds the zip, signs the zips and specs, and generates the packages.json file.
Example Use Cases
Scenario 1: Add a draft of a SFDC Lead Dump Hydrator pipeline
When the user clicks on the '+' button, the UI makes a call to get all the packages it can install:
Code Block |
---|
GET /v1/packages.json [ ..., { "name": "sfdc-lead-dump", "label": "SFDC Lead Dump", "canModifydescription": true"Reads SFDC data from a CDAP Stream, filters invalid }records, and dumps the data to a { CDAP Table.", "nameauthor": "configCask", "org": "Cask Data Inc.", "valueversion": "sfdc1.0.json0", // file in the archive"categories": [ "hydrator-pipelines" ] }, "canModify": false } ] } ] } |
...
...
] |
Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get the license for that package:
Code Block |
---|
GET /v1/packages/sfdc-lead-dump/1.0.0/license.txt
[ apache2 license ] |
The user accepts the conditions, and the UI makes a call to get the spec for that package:
Code Block |
---|
GET /groupsv1/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec.asc |
The UI also fetch the package archive and signature. It validates the package, and writes the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft
Code Block |
---|
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz.asc |
Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator draft.
Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.
When the user clicks on the '+' button, the UI makes a call:
Code Block |
---|
GET /groups [ { "name": "examples0/spec.json { "name": "sfdc-lead-dump", "label": "SFDC Lead Dump", "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.", "author": "Cask", "org": "Cask Data Inc.", "version": "1.0.1", "created": 1234567899, "changelog": "", "actions": [ { "type": "create_artifact", "arguments": [ { "name": "scope", "label": "Examples", "descriptionvalue": "Exampleuser" applications to get started with CDAP." }, { "name": "name", "value": "hydratorsfdc-plugins" }, "label": "Hydrator Plugins", { "description": "Plugins for Hydrator Pipelines." }"name": "version", ... ] |
to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Plugins', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Plugins' group:
Code Block |
---|
GET /groups/hydrator-plugins/packages [ ..., { "value": "1.0.0" }, { "name": "mysql-jdbc-driverconfig", "labelvalue": "MySQL JDBC Driver", "description": "JDBC Driver for MySQL databases.","sfdc-plugins.json" // file in the archive }, "author": "MySQL", { "org": "Oracle" }, ... ] |
Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get all versions of that package:
Code Block |
---|
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions [ { "name": "jar", "namevalue": "mysql-jdbc-driver"sfdc-plugins.jar" // file in the archive } ] }, "label{ "type": "create_app"MySQL, JDBC Driver", "descriptionarguments": "JDBC Driver for MySQL databases.",[ { "author": "MySQL", "orgname": "Oracleartifact", "version": "5.1.39", "createdvalue": 1234567899,{ "changelog": [ ], "dependenciesscope": { } }"system", ... ] |
The user decides to install the 5.1.38 version of the driver. The UI makes a call to get the spec, and to get the spec signature to make sure it is valid:
Code Block |
---|
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec { "name": "cdap-data-pipeline", "nameversion": "mysql-jdbc-driver",4.0.0" } "label": "MySQL JDBC Driver"}, "description": "JDBC Driver for MySQL databases.", "author": "MySQL", { "orgname": "Oraclename", "version": "5.1.39", "created": 1234567899, "actionsvalue": "SFDC [Lead Dump", { "typecanModify": "create_artifact",true "arguments": [ }, { "name": "scopeconfig", "value": "user",sfdc.json" // file in the archive } ] "canModify": false} ] }, { "name": "name", "value": "mysql-connector-java", "canModify": false }, { "name": "version", "value": "5.1.39", "canModify": false }, { "name": "parents", "value": "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]", "canModify": false }, { "name": "jar", "value": "mysql-connector-java-5.1.39-bin.jar", // file in the archive "canModify": false }, { } |
The UI also gets the spec signature to validate the spec:
Code Block |
---|
GET /v1/packages/sfdc-lead-dump/1.0.1/spec.json.asc |
The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft
Code Block |
---|
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip
GET /v1/packages/sfdc-lead-dump/1.0.1/archive.zip.asc |
Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator pipeline.
Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.
When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:
Code Block |
---|
GET /v1/packages.json
[
...,
{
"name": "mysql-jdbc-driver",
"label": "MySQL JDBC Driver",
"description": "JDBC Driver for MySQL databases.",
"author": "MySQL",
"org": "Oracle",
"version": "5.1.39",
"categories": [ "hydrator-plugins" ]
},
...
] |
Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:
Code Block |
---|
GET /v1/packages/mysql-jdbc-driver/5.1.39/license.txt
[ gpl license ] |
The user accepts the conditions, and the UI makes a call to get the spec for that package:
Code Block |
---|
GET /v1/packages/mysql-jdbc-driver/5.1.39/spec.json { "name": "mysql-jdbc-driver", "label": "MySQL JDBC Driver", "description": "JDBC Driver for MySQL databases.", "author": "MySQL", "org": "Oracle", "version": "5.1.39", "categories": [ "hydrator-plugins" ] "created": 1234567899, "actions": [ { "type": "create_artifact", "arguments": [ "name": "archivelink" { "valuename": "https://dev.mysql.com/downloads/file/?id=462849scope", "canModifyvalue": false"user" }, { "name": "archivesigname", "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.tar.gz" }, { "name": "pluginsversion", "value": {"5.1.39" }, { "parents": [ "cdap-data-pipeline[3.0.0,10.0.0]" ], "name": "externalArchive" "pluginsvalue": ["https://dev.mysql.com/downloads/file/?id=462849" }, { { "name" : "mysqlexternalArchiveSignature", "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz" "type" : "jdbc",}, { "classNamename" : "com.mysql.jdbc.DriverexternalArchiveJar", "description" : "Plugin for MySQL JDBC driver"value": "mysql-connector-java-5.1.39-bin.jar" }, } { ] "name": "config", }, "value": "mysql-connector-java-5.1.39.json" // file in the archive containing parents "canModify": falseand plugins } ] } ] } |
The UI also makes a call to get the spec signature to make sure it is valid:
Code Block |
---|
GET /v1/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc |
The UI then makes calls to get the archive and its signature to validate the archive, and unpack unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.
Code Block |
---|
GET /groups/hydrator-pluginsv1/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip.asc GET /groupsv1/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip |