Uploading a Plugin from Hub using cURL

This document describes how to upload a Plugin from Hub to Cloud Data Fusion without the need of downloading the plugin. The steps outlined illustrate how to deploy v0.14.13 of Google Cloud Platform plugins that are used to read from and write to various cloud storage systems. This package includes the Bigquery source and sink plugins, GCS sink and source plugins etc.

Before you begin

The steps outlined in this document require the use of jq command line library to parse a JSON file.

To install on Mac, you can use HomeBrew

brew install jq

1. Obtain plugin information from Hub

List available packages in the Hub:

https://hub.cdap.io/v2/packages.json

Once you have identified the plugin package you want, you can get the detail by going to:

https://hub.cdap.io/v2/packages/<PACKAGE NAME>/<PACKAGE VERSION>/spec.json

This will translate to hydrator-plugin-gcp-plugins as the package that contains all the Google Cloud Platform plugins. Inside the spec file, look at actions → arguments and obtain the version, name, jar file, and config file.

https://hub.cdap.io/v2/packages/hydrator-plugin-gcp-plugins/0.14.13/spec.json

{
  "specVersion": "1.0",
  "description": "Google Cloud Platform plugins to read from and write to various cloud storage systems.",
  "label": "Google Cloud Platform",
  "author": "Cask",
  "org": "Cask Data, Inc.",
  "cdapVersion": "[6.1.2,6.2.0-SNAPSHOT)",
  "created": 1613178518,
  "categories": [
    "hydrator-plugin"
  ],
  "actions": [
    {
      "type": "one_step_deploy_plugin",
      "label": "Deploy Google Cloud Platform plugins",
      "arguments": [
        {
          "name": "name",
          "value": "google-cloud",
          "canModify": false
        },
        {
          "name": "version",
          "value": "0.14.13",
          "canModify": false
        },
        {
          "name": "scope",
          "value": "user",
          "canModify": false
        },
        {
          "name": "config",
          "value": "google-cloud-0.14.13.json",
          "canModify": false
        },
        {
          "name": "jar",
          "value": "google-cloud-0.14.13.jar",
          "canModify": false
        }
      ]
    }
  ]
}

2. Set the environment variables for the Plugin

Example

export ARTIFACT_NAME=google-cloud
export ARTIFACT_VERSION=0.14.13
export ARTIFACT_CONFIG=google-cloud-0.14.13.json
export ARTIFACT_JAR=google-cloud-0.14.13.jar
export PACKAGE_NAME=hydrator-plugin-gcp-plugins
export PACKAGE_VERSION=0.14.13
export NAMESPACE=default

3. Upload Plugin JAR

Refer to CDAP reference on how to set the AUTH_TOKEN and CDAP_ENDPOINT variables.

curl https://hub.cdap.io/v2/packages/$PACKAGE_NAME/$PACKAGE_VERSION/$ARTIFACT_JAR | curl -X POST --data-binary @- \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/octet-stream" \
-H "Artifact-Version: ${ARTIFACT_VERSION}" \
-H "Artifact-Extends: system:cdap-data-pipeline[6.0.0-SNAPSHOT,7.0.0-SNAPSHOT)/system:cdap-data-streams[6.0.0-SNAPSHOT,7.0.0-SNAPSHOT)" \
"${CDAP_ENDPOINT}/v3/namespaces/${NAMESPACE}/artifacts/${ARTIFACT_NAME}"

3. Updating Plugin Properties (for Widgets and Docs)

curl https://hub.cdap.io/v2/packages/$PACKAGE_NAME/$PACKAGE_VERSION}/$ARTIFACT_CONFIG | jq ".properties" | \
curl -X PUT --data-binary @- \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/json" \
"${CDAP_ENDPOINT}/v3/namespaces/${NAMESPACE}/artifacts/${ARTIFACT_NAME}/versions/${ARTIFACT_VERSION}/properties"