Introduction 

CDAP hub serves as a central marketplace which can be accessed via CDAP UI “HUB” button. From the hub, the user can browse and search for plugins, directives etc and deploy them to their CDAP instances. At the moment, CDAP only supports installing resources from a single CDAP hub, which makes teams and organizations who want to support hosting custom resources such as drivers and plugins a difficult process.

Goals

Allows adding custom hubs to a CDAP instance in addition to the existing CDAP hub.  

User Stories 

• As an org admin, I want to make plugins developed internally available to all CDAP instances in the org or selected customers as a custom hub.

Design

The implementation of the existing CDAP hub feature heavily depends on the "market.base.url” property in cdap-site.xml file. The same value exists in the server/cdap.json file for development purpose. We use json representation in this doc for easier discussion. The value currently looks like the following:

{

  ...

  “market.base.url”: “https://hub.cdap.io/dev/v2”,

  ...

}

This field is exposed to both the node.js server and the client code running in the browser. On the node.js side, its value is used to assemble the final url to fetch hub resources such as packages.json, categories.json, icons and artifacts of packages. For example, to get the list of packages, the node.js server need to make a request to market.base.url + ‘/packages.json’. On the client side, it is used by market.js API layer. For image resources such as icons, client side directly make the request to fetch them instead of going through the node.js server.

To support multiple hubs, we propose to add a new field named “market.base.urls” in cdap-site.xml config file, as well as in the cdap.json file for development. The new field looks like the following:

{

  “market.base.urls”: "https://hub.cdap.io/dev/v2+https://yourmarketplaceurl.com/path+https://anothermarketplace.ca/path"

}

The users can add their own markets to CDAP by simply append their market host url to the “market.base.urls” string with one '+' as prefix. The + character is used as the separator for different urls. According to the current design, a valid market url must be encoded using the ASCII character set without any query parameter or hash. So using '+' as separator is safe. By default, the "market.base.urls" contains the default CDAP hub (the same as “market.base.url”).

Similar to existing pattern, this field is exposed to both the node.js server and the client side. On the client side, when making request to the node.js server (through API layer), a query parameter which specify the market url will be appended. For example instead of using “/packages.json”, “/packages.json?marketHost=https://yourmarketplaceurl.com/path” will be used for all market related requests. The API to get icons also take an optional field called marketHost, it will be used to decide where to fetch the image resources. If no marketHost is provided, it falls back to the old code path. On the node.js side, the query parameter from the client will be used as the market url to assemble the final resource url. It will be validated against the values from “market.base.urls” field to make sure that the url is an expected and valid market host. For example, if marketHost is “https://evil.com”, and the “https://evil.com” does not exist in market.base.urls, the nodejs should reject the request from client side for security reason.

We also propose to add a getMetadata API in market API layer. It will try to fetch metadata information about the market from marketUrl/metadata.json. The json file should only contains the name of the market for now. The market name will be used to as the text displayed in a horizontal market tab (see screenshot below). If getMetaAPI failed, the domain name will be used as fallback. The metadata.json should look like this:

With this design, we expect all custom hub structured exactly the same as CDAP’s hub. This means, it should look like the following structure:

.

+ -- metadata.json

+ -- packages.json

+ -- categories.json

+ -- packages

      + -- package a

            + --  version 1

                   + --  spec.json

                   + --  icon.png

                   + --  *.jar

           + --  version 2

     + --  package b

     ...

The common parent path (root) to these files should be provided as the url field in cdap-site.xml config file above.

After everything is set up correctly, customers of the custom hubs should now see a horizontal tab when they open the CDAP hub. Each tab represents one hub and the name provided in the cdap-site.xml conf file will be displayed as the tab name. When customer selected a hub, the client side will kick off resource fetching for the selected hub and render the UI once resources are ready.

UI Impact or Changes

• If there are more than 1 hub in “market.base.urls”, a horizontal tab will be showed at the top of the existing hub UI to allow user switch between hubs. See screenshot below

  
  

Security Impact 

Client might send random market url to the node.js server using marketHost parameter. On the node.js server, we validate the url from client side before proceed.

Impact on Infrastructure Outages 

If a hub becomes inaccessible (i.e. the admin moved them to a new location or permission issue) or miss configured, an error message (see here) will be shown in the UI for the particular hub. Note this should not affect user to access other hubs.

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work