This document describes how to upload a Plugin from Hub to Cloud Data Fusion without the need of downloading the plugin. The steps outlined illustrate how to deploy v0.14.13 of Google Cloud Platform plugins that are used to read from and write to various cloud storage systems. This package includes the Bigquery source and sink plugins, GCS sink and source plugins etc.
Before you begin
The steps outlined in this document require the use of jq command line library to parse a JSON file.
...
Code Block |
---|
brew install jq |
1.
...
Obtain plugin information from Hub
List available packages in the Hub:
...
Code Block |
---|
https://hub.cdap.io/v2/packages/<PACKAGE NAME>/<PACKAGE VERSION>/spec.json |
inside This will translate to hydrator-plugin-gcp-plugins as the package that contains all the Google Cloud Platform plugins. Inside the spec file, look at actions → arguments and obtain the version, name, jar file, and config file.
example:
Code Block |
---|
ARTIFACT NAME: postgresql-plugin
ARTIFACT VERSION: 1.1.0
ARTIFACT CONFIG: postgresql-plugin-1.1.0.json
ARTIFACT JAR: postgresql-plugin-1.1.0.jar |
2. Upload Plugin JAR
Example:
Expand |
---|
title | https://hub.cdap.io/v2/packages/hydrator-plugin-gcp-plugins/0.14.13/spec.json |
---|
|
Code Block |
---|
{
"specVersion": "1.0",
"description": "Google Cloud Platform plugins to read from and write to various cloud storage systems.",
"label": "Google Cloud Platform",
"author": "Cask",
"org": "Cask Data, Inc.",
"cdapVersion": "[6.1.2,6.2.0-SNAPSHOT)",
"created": 1613178518,
"categories": [
"hydrator-plugin"
],
"actions": [
{
"type": "one_step_deploy_plugin",
"label": "Deploy Google Cloud Platform plugins",
"arguments": [
{
"name": "name",
"value": "google-cloud",
"canModify": false
},
{
"name": "version",
"value": "0.14.13",
"canModify": false
},
{
"name": "scope",
"value": "user",
"canModify": false
},
{
"name": "config",
"value": "google-cloud-0.14.13.json",
"canModify": false
},
{
"name": "jar",
"value": "google-cloud-0.14.13.jar",
"canModify": false
}
]
}
]
}
|
|
2. Set the environment variables for the Plugin
Example
Code Block |
---|
export ARTIFACT_NAME=google-cloud
export ARTIFACT_VERSION=0.14.13
export ARTIFACT_CONFIG=google-cloud-0.14.13.json
export ARTIFACT_JAR=google-cloud-0.14.13.jar
export PACKAGE_NAME=hydrator-plugin-gcp-plugins
export PACKAGE_VERSION=0.14.13
export NAMESPACE=default |
3. Upload Plugin JAR
Refer to CDAP reference on how to set the AUTH_TOKEN and CDAP_ENDPOINTÂ variables.
Code Block |
---|
curl https://hub.cdap.io/v2/packages/<PACKAGE NAME>/<PACKAGE VERSION>/<ARTIFACT JAR>$PACKAGE_NAME/$PACKAGE_VERSION/$ARTIFACT_JAR | curl -X POST --data-binary @- \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/octet-stream" \
-H "Artifact-Version: <ARTIFACT VERSION>${ARTIFACT_VERSION}" \
-H "Artifact-Extends: system:cdap-data-pipeline[6.0.0-SNAPSHOT,7.0.0-SNAPSHOT)/system:cdap-data-streams[6.0.0-SNAPSHOT,7.0.0-SNAPSHOT)" \
"${CDFCDAP_ENDPOINT}/v3/namespaces/<NAMESPACE>${NAMESPACE}/artifacts/<ARTIFACT NAME>${ARTIFACT_NAME}" |
3. Updating Plugin Properties (for Widgets and Docs)
Code Block |
---|
curl https://hub.cdap.io/v2/packages/<PACKAGE NAME>/<PACKAGE VERSION>/<ARTIFACT CONFIG>$PACKAGE_NAME/$PACKAGE_VERSION}/$ARTIFACT_CONFIG | jq ".properties" | \
curl -X PUT --data-binary @- \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/json" \
"${CDFCDAP_ENDPOINT}/v3/namespaces/<NAMESPACE>${NAMESPACE}/artifacts/<ARTIFACT NAME>${ARTIFACT_NAME}/versions/<ARTIFACT VERSION>${ARTIFACT_VERSION}/properties"
|
Related articles