Page Comparison

...

Code Block

{
  "type": "create_artifact" | "create_app" | "create_stream" | "create_dataset" | "create_hydrator_draft",
  "arguments": [
    {
      "name": [argument name],
      "value": [argument value],
      "canModify": true | false (defaults to false)
    }
  ]
}

Some arguments can be modified by users in the resulting wizard. For example, the name of an application may be a field that the user should be able to edit.

...

name	description	required?	default
name	artifact name	yes
jar	name of jar file in package archive	yesno (if using externalArchive)
archivelinkexternalArchive	link to download 3rd party archive	no	none
archivesigexternalArchiveSignature	link to get 3rd party archive signature	no	none
externalArchiveJar	path of the jar file in the external archive	no	none
scope	artifact scope (implies API to add system artifacts is added in 4.0)	no	user
version	artifact version to pass as Artifact-Version header	no	none
parentsconfig	config file contains artifact parents to pass as Artifact-Extends header	no	none	plugins	artifact plugins to pass as Artifact-Plugins header	, plugins, and properties	no	none

create_app

Results in a call to http://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#create-an-application

...

name	description	required?	default
name	dataset name	yes
type	dataset type	yes
description	dataset description	no	empty
properties	json map of dataset properties	no	empty

create_hydrator_draft

Results in whatever the UI does to create a draft

name	description	required?
name	pipeline name	yes
artifact	scope, name, version of the artifact to create the app with	yes
config	pipeline config (file in the package archive)	yes

Dependencies

...

Dependencies

Packages will be able to specify dependencies on other packages.

...

Code Block

GET /4.0.0/packages.json
[
  ...,
  {
    "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
    "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
  },  "version": "1.0..
]

...

0",
    "categories": [ "hydrator-pipelines" ]
  },
  ...
]

Among that list is version 1.0.1 of the 'SFDC Lead Dump' package, which the user clicks on. The UI makes a call to get all versions of the license for that package:

Code Block
GET /groups/hydrator-plugins4.0.0/packages/sfdc-lead-dump/versions1.0.0/license.txt [ apache2 { license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /4.0.0/packages/sfdc-lead-dump/1.0.0/spec.json
{
  "name": "sfdc-lead-dump",
    "label": "SFDC Lead Dump",
 
  "description": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
    "author": "Cask",
    "org": "Cask Data Inc.",
    "version": "1.0.1",
    "created": 1234567899,
  "changelog": "",
  "changelogactions": [
    {
 "fixed  a small parsing bug"type"
  : "create_artifact",
 ],     "dependenciesarguments": {[
       "cdap": {
          "minVersionname": "4.0.0scope",
          "maxVersionvalue": "4.1.0user"
        },
     }   },{
  ...
]

It defaults to the most recent version that is compatible with the version of CDAP that is running. The user decides to install the package, so the UI makes a call to get the package spec:

Code Block

GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec
{        "name": "sfdc-lead-dumpname",
  "label": "SFDC Lead Dump",        "descriptionvalue": "Reads SFDC data from a CDAP Stream, filters invalid records, and dumps the data to a CDAP Table.",
  "author": "Cask"sfdc-plugins"
        },
        {
          "name": "version",
  "org": "Cask Data Inc.",        "versionvalue": "1.0.10",
  "created": 1234567899,   "changelog": [ },
     "fixed a small parsing bug"   {
       ],   "actionsname": [
"config",
   {       "typevalue": "create_artifact",sfdc-plugins.json" // file in the archive
       "arguments": [ },
        {
          "name": "scopejar",
          "value": "usersfdc-plugins.jar", // file in the archive
     "canModify": false  }
      },]
    },
    {
      "type": "create_app",
      "namearguments": "name",[
        {
          "valuename": "sfdc-pluginsartifact",
          "canModifyvalue": false{
        },    "scope": "system",
   {           "name": "versioncdap-data-pipeline",
            "valueversion": "14.0.0",
          "canModify":}
false         },
        {
          "name": "parentsname",
          "value": "system:cdap-data-pipeline[4.0.0,4.1.0)SFDC Lead Dump",
          "canModify": falsetrue
        },
        {
          "name": "jarconfig",
          "value": "sfdc-plugins.jarjson", // file in the archive
        }
 "canModify": false    ]
    }
   
  ]
    },
    {
      "type": "create_hydrator_draft",
      "arguments": [
        {
          "name": "artifact",
          "value": {
            "scope": "system",
            "name": "cdap-data-pipeline",
            "version": "4.0.0"
          },
          "canModify": false
        },
        {
          "name": "name",
          "value": "SFDC Lead Dump",
          "canModify": true
        },
        {
          "name": "config",
          "value": "sfdc.json", // file in the archive
          "canModify": false
        }
      ]
    }
  ]
}

The UI also fetches the spec signature and uses the public key to validate the spec:

Code Block
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/spec.asc

The UI also fetch the package archive and signature. It validates the package, and writes the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz GET /groups/hydrator-pipelines/packages/sfdc-lead-dump/versions/1.0.1/archive.tgz.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator draft.

Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.

When the user clicks on the '+' button, the UI makes a call:

Code Block

GET /groups
[
  {
    "name": "examples",
    "label": "Examples",
    "description": "Example applications to get started with CDAP."
  },
  {
    "name": "hydrator-plugins",
    "label": "Hydrator Plugins",
    "description": "Plugins for Hydrator Pipelines."
  },
  ...
]

to display all the different types of things the user can add in the CDAP marketplace. Among that list is 'Hydrator Plugins', which the user clicks on. The UI makes another call to list the packages in the 'Hydrator Plugins' group:

Code Block
GET /groups/hydrator-plugins/packages [ ..., { "name": "mysql-jdbc-driver", "label": "MySQL JDBC Driver", "description": "JDBC Driver for MySQL databases.", "author": "MySQL", "org": "Oracle" }, ... ]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get all versions of that package:

Code Block

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions
[
  {
    "name": "mysql-jdbc-driver",
    "label": "MySQL JDBC Driver",
    "description": "JDBC Driver for MySQL databases.",
    "author": "MySQL",
    "org": "Oracle",
    "version": "5.1.39",
    "created": 1234567899,
    "changelog": [ ],
    "dependencies": { }
  },
  ...
]

The user decides to install the 5.1.38 version of the driver. The UI makes a call to get the spec, and to get the spec signature to make sure it is valid:

Code Block

GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec
{    
  "name": "mysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
  "description": "JDBC Driver for MySQL databases.",
  "author": "MySQL",
  "org": "Oracle",
  "version": "5.1.39",
  "created": 1234567899,
  "actions": [
    {
      "type": "create_artifact",
      "arguments": [
     
}

The UI also gets the spec signature to validate the spec:

Code Block
GET /4.0.0/packages/sfdc-lead-dump/1.0.1/spec.json.asc

The UI also fetches the package archive and signature. It validates the package, and unzips the archive to a local temporary directory so that it can use its resources to create the plugins artifact and create the hydrator draft

Code Block
GET /4.0.0/packages/sfdc-lead-dump/1.0.1/archive.tgz GET /4.0.0/packages/sfdc-lead-dump/1.0.1/archive.tgz.asc

Based on the package spec, the UI can setup the relevant wizards and make the relevant CDAP calls to first create the plugin artifact, and next create the Hydrator pipeline.

Scenario 7: Add MySQL jdbc driver as a Hydrator plugin.

When the user clicks on the '+' button, the UI makes a call to list all packages that can be added to CDAP:

Code Block

GET /4.0.0/packages.json
[
  ...,
  {
          "name": "scopemysql-jdbc-driver",
  
       "valuelabel": "userMySQL JDBC Driver",
 
        "canModifydescription": false"JDBC Driver for       },
        {
    MySQL databases.",
     "nameauthor": "nameMySQL",
    "org": "Oracle",
    "valueversion": "mysql-connector-java5.1.39",
    "categories": [ "hydrator-plugins" ]
  "canModify": false},
         },
        {
 ...
]

Among the list is the MySQL JDBC Driver, which the user clicks on. The UI makes a call to get the license for that package:

Code Block
GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/license.txt [ gpl license ]

The user accepts the conditions, and the UI makes a call to get the spec for that package:

Code Block

GET /4.0.0/packages/mysql-jdbc-driver/5.1.39/spec.json
{    
    "name": "versionmysql-jdbc-driver",
  "label": "MySQL JDBC Driver",
    "valuedescription": "5.1.39",
   JDBC Driver for MySQL databases.",
  "author": "MySQL",
  "canModifyorg": false
        },
        {
       "Oracle",
  "nameversion": "parents5.1.39",
  "categories": [ "hydrator-plugins" ]
  "created": 1234567899,
  "valueactions": "system:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]",[
    {
      "canModifytype": false"create_artifact",
      "arguments": [
},         {
          "name": "jarscope",
          "value": "mysql-connector-java-5.1.39-bin.jar", // file in the archiveuser"
        },
        {
          "canModifyname": false"name",
        },  "value": "mysql-connector-java"
     {   },
       "name": "archivelink" {
          "valuename": "https://dev.mysql.com/downloads/file/?id=462849version",
          "canModifyvalue": false"5.1.39"
        },
        {
          "name": "archivesigparents",
          "value": "https://dev.mysql.com/downloads/gpg/?file=mysql-connector-java-5.1.39.tar.gzsystem:cdap-data-pipeline[3.0.0,10.0.0]/system:cdap-data-streams[3.0.0,10.0.0]"
        },
        {
          "name": "pluginsexternalArchive",
          "value": {
            "parents": [ "cdap-data-pipeline[3.0.0,10.0.0]" ],"https://dev.mysql.com/downloads/file/?id=462849"
        },
    "plugins": [   {
           {
     "name": "externalArchiveSignature",
          "namevalue" : "https://dev.mysql",.com/downloads/gpg/?file=mysql-connector-java-5.1.39.zip.gz"
        },
       "type" : "jdbc", {
                "className" "name": "com.mysql.jdbc.DriverexternalArchiveJar",
             "value": "mysql-connector-java-5.1.39.zip.gz"
  "description" : "Plugin for MySQL JDBC driver"},
        {
     }     "name": "config",
      ]    "value": "mysql-connector-java-5.1.39.json" // file in the archive
},        }
  "canModify": false   ]
     }
      ]
    }
  ]
}]
}

The UI also makes a call to get the spec signature to make sure it is valid:

Code Block
GET /groups/hydrator-plugins/packages/mysql-jdbc-driver/versions/5.1.39/spec.asc

The UI then makes calls to get the archive and its signature to validate the archive, and unpack unzip it in a local directory. It uses the jar and json config file contained in the archive to make a request to add the artifact to cdap.

Code Block
GET /groups/hydrator-plugins4.0.0/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip.asc GET /groups/hydrator-plugins/4.0.0/packages/mysql-jdbc-driver/versions/5.1.3839/archive.tgzzip

Versions Compared

Old Version 35

New Version 36

Key

create_app

create_hydrator_draft

Dependencies

Dependencies