• Scenario 1: Specifying Capability

  • Scenario 1.1

Alice is a CDAP Plugin developer who is developing a CDAP Dataset plugin (transactional). Her plugin is supported only in transactional environment. She will like to specify this in her plugin so that pipeline developer don’t use her plugin in other modes.

• • Scenario 1.2

Alice is also developing an Action plugin which store some state information in CDAP Dataset. Since her action plugin uses CDAP Dataset it can only run in native mode. She will like to specify this in her plugin so that pipeline developer don’t use her plugin in other modes.

• • Scenario 1.3

Alice is CDAP plugin developer who is developing a Spark ML transform which is uses libraries available only in Spark 2 and she will like to specify her plugin is only compatible with Spark 2.

• • Scenario 1.4

Alice is a CDAP Plugin developer who is developing a PII compliant plugin and she will like to specify that her plugin is PII complaint so that when she deploys her plugin in a CDAP instance which only allow PII complaint plugins to run her plugin can be run and be used by pipeline developers.

• Scenario 2: Plugin Filtering

• Scenario 2.1

Bob is a data analyst who is evaluating CDAP. He is running his CDAP in a particular environment and he sees a lot of plugin which does not seem compatible with his environment. He will like to be able to filter plugins on compatibility to see only the plugins which is compatible with his environment.

• • Scenario 2.2

Eve is a CDAP administrator, who is setting up a CDAP instance in cloud. She will like to enforce that only plugins which are capable of running in cloud are available to pipeline developer for use.

• • Scenario 2.3

Eve is a CDAP administrator, who is trying to set up a CDAP environment in production for data processing. Eve’s organization has strict compliance requirement and she wants to only allow plugins which meet certain compliance to be used by the data analyst in her organization. Furthermore, she does not want any data analyst to be able to override her settings and be able to run non-compliant plugins.

• Scenario 3: Failing Early and Gracefully

• Scenario 3.1

Bob is trying to develop a pipeline to process some data which is stored in CDAP Table. He builds a pipeline with the appropriate plugin and configuration and the pipeline fails at runtime with a lot of cryptic error messages in logs. Bob rechecks his plugin configurations and tries to debug the issue but he is not able to run the pipeline successfully. Disappointed with the platform Bob reaches out to CDAP support group for help. After some back and forth Bob gets to know that he was running the plugin in cloud mode and to run this pipeline he will need to set the correct compute profile during runtime. It makes sense to him but he wonders only if the log error messages would have pointed it out, he could have easily corrected it by himself saving the time spent in support.

• • Scenario 3.2

Bobs exported a plugin which was sent to him by another pipeline developer and tries to run it. The pipeline fails for him but works perfectly fine for his colleague. Bobs tries to debug the issue by looking into the logs but he is again greeted by cryptic error messages. He reaches out to CDAP support and was told that he is running his pipeline in incorrect mode. He gets really furious as why CDAP logs does not show any information for such a common problem.

/**
 * Contains information about a plugin class.
 */
@Beta
public class PluginClass {

  private final String type;
  private final String name;
  private final String description;
  private final String className;
  private final String configFieldName;
  private final Map<String, PluginPropertyField> properties;
  private final Set<String> endpoints;
  private final Set<String> capability; // all the capability of this plugin
}

Storage

We need to store the capability information at plugin level as one artifact can have n numbers of plugins and each one of them will have their own capability information.

Approach 1: Artifact Store (Selected)

We are storing the plugin capability information in PluginClass which is contained in ArtifactMeta. Hence, we can easily store the capability information of a plugin in the ArtifactStore as a part of ArtifactMeta itself. This will allow us to store all the plugin information in one store.

Approach 2: Metadata (Considered)

The capability information can also be stored as a system metadata of the Plugin by the ArtifactSystemMetadataWriter. Since Plugin is not an EntityId in CDAP we will use Metadata systems capability to store metadata for custom entities where Plugin will be a custom entity under Artifact.

The custom entity hierarchy will be as follows:

namespace=<namespace-name> | artifact=<artifact-name> | version=<artifact-version> | plugin=<plugin-name>

Note: | and = are just used as a separator here for readability. In actual serialized form we use byte-length encoding.

This capability information will be stored as metadata property where the key will be ‘capability’ and value list of unique comma separated string representing capabilities.

capability = mode_sandbox, mode_cloud, pii

Note: = and , is our standard key-value and individual value separator in Metadata storage

Comparison

The below table shows the comparison between the two approaches

<property>
	<name>plugin.required.capabilities</name>
	<value>mode_cloud</value>
	<description>
		Comma separated list of capabilities values which are required by default. If a system level capability category is undefined no capability is required in that category and all plugins for that category will be displayed/enabled. If this property is undefined or is empty no capability is required in any of the system defined categories.
    </description>
</property>

(mode_cloud && spark_2) || (mode_native && spark_1)

public interface Provisioner {

  /**
   * Returns all the capabilities which plugins must have to be able to run
   *
   * @return a {@link Set} of {@link String} which represents Capabilities
   */
  Set<String> getRequiredCapabilities();
  ...
  ...
}

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capability": [
     "mode_cloud",
     "spark_2"
   ]
 },
 {
   "name": "Plugin2",
   "type": "dummy",
   "description": "This is plugin2",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin2",
   "artifact": {
     "name": "plugins",
     "version": "2.0.0",
     "scope": "USER"
   },
   "capability": [
     "mode_cloud",
     "spark_1"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capability=spark_2

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capabilities": [
     "mode_cloud",
     "spark_2"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}

GET /namespaces/{namespace-id}/capabilities

[
 "mode_cloud",
 "spark_2",
 "spark_1",
 "mode_native"
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capability=spark_2

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capabilities": [
     "mode_cloud",
     "spark_2"
   ]
 }
]

Comparison

To understand how the configuration and filtering will work in real world let us consider few use cases which we know of and see how they can be addressed through the above design.

Cloud

CDAP is running in a cloud environment and we only want to allow plugins capable of running in cloud to be displayed

CDAP is running in a hadoop environment and we only want to allow plugins capable of running in hadoop and outside cloud or server connection is restricted.

In-Prem Hadoop

CDAP is running in hadoop environment and we only want to allow plugins capable of running in hadoop and cloud

Sandbox

CDAP is running in sandbox and we want to allow plugin which are capable of running in native or cloud

Note: When plugin.required.capabilities is empty it means the instance does not defines any capability to be required for any category and hence all plugins will be shown.

Sandbox: Spark 2

CDAP is running in sandbox and we want to allow plugin which are capable of running with spark 2

Note: Here plugin.required.capabilities does not specify any requirement for Mode which means the system does not defines any capability for mode to be required and hence all plugins which support any mode are a candidate for being display. Although it does define a Spark requirement so only plugins which is compatible with spark 2 will be show.

Sandbox: Compliance Required

CDAP is running in sandbox and we want to allow plugin which are capable of running in native or cloud mode but also want to satisfy a compliance need and hence plugins must have PII capability

Cloud: Compliance Required

CDAP is running in cloud and we want to allow only plugins which are capable of running in cloud and is PII compliant

Preview

A pipeline preview runs in native (local machine) JVM. Hence, any plugin which is capable of running in Native environment is capable of preview. If a plugin which is not capable of running in Native is run in a preview then that preview operation will failed with incapability as soon as the preview is started and before the actual preview process runs.

Failing Early and Gracefully

Filtering incapable plugins does not guarantee that a pipeline containing an incapable plugin will not be run in the system. This might happen because of various reason for example, a pipeline was created before a capability was required i.e. the capability was added as a requirement in the plugin.required.capabilities configuration after the pipeline deployment.

Whenever we encounter a plugin which is incapable to run we will like to fail early and gracefully. Ideally, we will like to fail as early as possible. We will fail the plugin at different stages: 

1. Deploy time: Since plugins which are incapable to run will be filtered out, a user will not be able to create a pipeline using them. If a user imports a pipeline json which contains an incapable plugin then the deployment will fail with Artifact/Plugin not found exception. Note: The exception here will not be Incapability exception since that will expose the existence of of the plugin in the system and just being filtered outwhich we do not want to expose.
2. Run Time (Before run): In addition to filtering a plugin we will also eno

1. be doing checks before provisioning. This will fail the pipeline if any of the plugin is not capable to run in the provisioned environment. For example running a plugin which AWS which is not capable to run in AWS and can only run in Azure.

Whenever we will fail a pipeline due to incapability we will surface this information to user through logs and also an error message in UI.