• Scenario 1: Specifying Capability

  • Scenario 1.1

Alice is a CDAP Plugin developer who is developing a CDAP Dataset plugin (transactional). Her plugin is supported only in transactional environment. She will like to specify this in her plugin so that pipeline developer don’t use her plugin in other modes.

• • Scenario 1.2

Alice is also developing an Action plugin which store some state information in CDAP Dataset. Since her action plugin uses CDAP Dataset it can only run in native environment. She will like to specify this in her plugin so that pipeline developer don’t use her plugin in other environment.

• • Scenario 1.3

Alice is CDAP plugin developer who is developing a Spark ML transform which is uses libraries available only in Spark 2 and she will like to specify her plugin is only capable of running on Spark 2.

• • Scenario 1.4

Alice is a CDAP Plugin developer who is developing a PII capable plugin and she will like to specify that her plugin is PII capable so that when she deploys her plugin in a CDAP instance which only allow PII capable plugins to run her plugin can be run and be used by pipeline developers.

• Scenario 2: Plugin Filtering

• Scenario 2.1

Bob is a data analyst who is evaluating CDAP. He is running his CDAP in a particular environment and he sees a lot of plugin which does not seem capable to run in his environment. He will like to be able to filter plugins on capability to see only the plugins which is capable with his environment.

• • Scenario 2.2

Eve is a CDAP administrator, who is setting up a CDAP instance in cloud. She will like to enforce that only plugins which are capable of running in remote environment are available to pipeline developer for use.

• • Scenario 2.3

Eve is a CDAP administrator, who is trying to set up a CDAP environment in production for data processing. Eve’s organization has strict compliance requirement and she wants to only allow plugins which meet certain compliance to be used by the data analyst in her organization. Furthermore, she does not want any data analyst to be able to override her settings and be able to run non-compliant plugins.

• Scenario 3: Failing Early and Gracefully

• Scenario 3.1

Bob is trying to develop a pipeline to process some data which is stored in CDAP Table. He builds a pipeline with the appropriate plugin and configuration and the pipeline fails at runtime with a lot of cryptic error messages in logs. Bob rechecks his plugin configurations and tries to debug the issue but he is not able to run the pipeline successfully. Disappointed with the platform Bob reaches out to CDAP support group for help. After some back and forth Bob gets to know that he was running the plugin remotely and to run this pipeline he will need to set the correct compute profile during runtime. It makes sense to him but he wonders only if the log error messages would have pointed it out, he could have easily corrected it by himself saving the time spent in support.

• • Scenario 3.2

Bobs exported a plugin which was sent to him by another pipeline developer and tries to run it. The pipeline fails for him but works perfectly fine for his colleague. Bobs tries to debug the issue by looking into the logs but he is again greeted by cryptic error messages. He reaches out to CDAP support and was told that he is running his pipeline in incorrect mode. He gets really furious as why CDAP logs does not show any information for such a common problem.

/**
 * Contains information about a plugin class.
 */
@Beta
public class PluginClass {

  private final String type;
  private final String name;
  private final String description;
  private final String className;
  private final String configFieldName;
  private final Map<String, PluginPropertyField> properties;
  private final Set<String> endpoints;
  private final Set<String> capability; // all the capability of this plugin
}

Storage

Capability information will be stored at plugin level as one artifact can have n numbers of plugins and each one of them will have their own capability information.

Approach 1: Artifact Store (Selected)

Plugin capability information will be stored in PluginClass which is contained in ArtifactMeta. Hence, capability information of a plugin can easily be stored in the ArtifactStore as a part of ArtifactMeta itself. This allow us to store all the plugin information in one store.

Approach 2: Metadata (Considered)

The capability information can also be stored as a system metadata of the Plugin by the ArtifactSystemMetadataWriter. Since Plugin is not an EntityId in CDAP, Metadata systems capability to store metadata for custom entities can be used where the Plugin will be a custom entity under Artifact.

The custom entity hierarchy will be as follows:

namespace=<namespace-name> | artifact=<artifact-name> | version=<artifact-version> | plugin=<plugin-name>

Note: | and = are just used as a separator here for readability. In actual serialized form byte-length encoding is used.

This capability information will be stored as metadata property where the key will be ‘capability’ and value list of unique comma separated string representing capabilities.

capability = remote, native, pii

Note: = and , is our standard key-value and individual value separator in Metadata storage

Comparison

The below table shows the comparison between the two approaches

<property>
	<name>plugin.required.capabilities</name>
	<value>remote</value>
	<description>
		Comma separated list of capabilities values which are required by default. If system level capability is undefined no capability is required and all plugins will be displayed/enabled.
    </description>
</property>

(remote && spark2) || (native && spark1)

public interface Provisioner {

  /**
   * @returns a Set of capabilities which are explicitly defined to be supported by the provisioner or an empty set if the provisioner does not define any explicit capabilities
   */
  void Set<String> getCapabilities();
  ...
  ...
}

public interface ProgramSpecification {
  /**
   * @returns the requirements for this program or an empty set if the program does not have any specific requirements defined
   */
  Set<String> getRequirements();
}

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capability": [
     "remote",
     "spark2"
   ]
 },
 {
   "name": "Plugin2",
   "type": "dummy",
   "description": "This is plugin2",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin2",
   "artifact": {
     "name": "plugins",
     "version": "2.0.0",
     "scope": "USER"
   },
   "capability": [
     "remote",
     "spark1"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capabilities=spark2

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capabilities": [
     "remote",
     "spark2"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}

GET /namespaces/{namespace-id}/capabilities

[
 "remote",
 "spark2",
 "spark1",
 "native"
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capabilities=spark2

[
 {
   "name": "Plugin2Plugin1",
   "type": "dummy",
   "description": "This is plugin2plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin2Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "21.0.0",
     "scope": "USER"
   },
   "capabilitycapabilities": [
     "mode_cloudremote",
     "spark_1spark2"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capability=spark_2

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capabilities": [
     "mode_cloud",
     "spark_2"
   ]
 }
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}

GET /namespaces/{namespace-id}/capabilities

[
 "mode_cloud",
 "spark_2",
 "spark_1",
 "mode_native"
]

GET /namespaces/{namespace-id}/artifacts/{artifact-name}/versions/{artifact-version}/extensions/{plugin-type}?capability=spark_2

[
 {
   "name": "Plugin1",
   "type": "dummy",
   "description": "This is plugin1",
   "className": "co.cask.cdap.internal.app.runtime.artifact.plugin.Plugin1",
   "artifact": {
     "name": "plugins",
     "version": "1.0.0",
     "scope": "USER"
   },
   "capabilities": [
     "mode_cloud",
     "spark_2"
   ]
 }
]

Comparison

Approach

Pros

Cons

Approach 1

• Does not require UI to make an additional call to get a list  of all compat options

• Will be extra work on the client/ui side to parse all the compat options to process/render it.

Approach 2

• Allows a client to query for all the compatibility options present in the system.

• Easier for CDAP UI to render a selector widget to select/deselect compatibility options

• CDAP UI will have to make an additional call to fetch this information before fetching plugins

Filtering Examples

To understand how the configuration and filtering will work in real world let us consider few use cases which we know of and see how they can be addressed through the above design.

Cloud

Comparison

To understand how the configuration and filtering will work in real world let us consider few use cases which us known so far and see how they can be addressed through the above design.

Cloud

CDAP is running in a cloud environment and an administrator only want to allow plugins capable of running in cloud to be displayed

CDAP is running in a hadoop environment and administrator only want to allow plugins capable of running in hadoop and outside cloud or server connection is restricted.

In-Prem Hadoop

CDAP is running in hadoop environment and administrator only want to allow plugins capable of running in hadoop and cloud

Sandbox

CDAP is running in sandbox and administrator want to allow plugin which are capable of running in native or cloud

Note: When plugin.required.capabilities is empty it means the instance does not defines any capability to be required for any category and hence all plugins will be shown. Although if a plugin is used in incompatible profile then the pipeline will fail before provisioning. Consider a case where CDAPTableSource is being used in a pipeline and the profile is set to EMR Provisioner. EMR Provisioner will do a check that all the plugins define their capability with "emr". Since the requirement will not be meet the pipeline will marked failed immediately due to incompatibility. Although if the pipeline had all "emr" compatible plugins like AWSS3Source then the requirement will be meet, the cluster will be provisioned and the pipeline will be run. 

Sandbox: Spark 2

CDAP is running in sandbox and administrator want to allow plugin which are capable of running with spark 2

Note: Here plugin.required.capabilities does not specify any requirement for Mode which means the system does not defines any capability for mode to be required and hence all plugins which support any mode are a candidate for being display. Although it does define a Spark requirement so only plugins which is compatible with spark 2 will be show.

Sandbox: Compliance Required

CDAP is running in sandbox and administrator want to allow plugin which are capable of running in native or cloud mode but also want to satisfy a compliance need and hence plugins must have PII capability

Cloud: Compliance Required

CDAP is running in cloud and administrator want to allow only plugins which are capable of running in cloud and is PII compliant

Visible

In-Prem Hadoop

CDAP is running in a hadoop environment and we only want to allow plugins capable of running in hadoop and outside cloud or server connection is restricted.

plugin.required.capability

CDAP Table Source

@Capability(Mode.NATIVE, Spark.V2, "PII")

BigTable Source

@Capability(Mode.CLOUD, "PII")

AWS S3 Source

@Capability(Mode.CLOUD, Mode.NATIVE)

mode_native

Visible

Filtered out

Visible

In-Prem Hadoop

CDAP is running in hadoop environment and we only want to allow plugins capable of running in hadoop and cloud

plugin.required.capability

CDAP Table Source

@Capability(MODE.NATIVE, PII)

BigTable Source

@Capability(MODE.CLOUD, PII)

AWS S3 Source

@Capability(MODE_CLOUD, MODE.NATIVE)

mode_native, mode_cloud

Filtered out

Filtered out

Visible

Sandbox

CDAP is running in sandbox and we want to allow plugin which are capable of running in native or cloud

plugin.required.capability

CDAP Table Source

@Capability(Mode.NATIVE, Spark.V2, "PII")

BigTable Source

@Capability(Mode.CLOUD, "PII")

AWS S3 Source

@Capability(Mode.CLOUD, Mode.NATIVE)

Visible

Visible

Visible

Note: When plugin.required.capabilities is empty it means the instance does not defines any capability to be required for any category and hence all plugins will be shown.

Sandbox: Spark 2

CDAP is running in sandbox and we want to allow plugin which are capable of running with spark 2

plugin.required.capability

CDAP Table Source

@Capability(Mode.NATIVE, Spark.V2, "PII")

BigTable Source

@Capability(Mode.CLOUD, "PII")

AWS S3 Source

@Capability(Mode.CLOUD, Mode.NATIVE)

spark_2

Visible

Filtered out

Filtered out

Note: Here plugin.required.capabilities does not specify any requirement for Mode and hence it means the instance does not defines any capability for mode to be required and hence all plugins which support any mode are a candidate to be displayed. Although it does define a Spark requirement so only plugins which is compatible with spark 2 will be show.

Sandbox: Compliance Required

CDAP is running in sandbox and we want to allow plugin which are capable of running in native or cloud mode but also want to satisfy a compliance need and hence plugins must have PII capability

plugin.required.capability

CDAP Table Source

@Capability(Mode.NATIVE, Spark.V2, "PII")

BigTable Source

@Capability(Mode.CLOUD, "PII")

AWS S3 Source

@Capability(Mode.CLOUD, Mode.NATIVE)

PII

Visible

Visible

Filtered out

Cloud: Compliance Required

CDAP is running in cloud and we want to allow only plugins which are capable of running in cloud and is PII compliant

plugin.required.capability

CDAP Table Source

@Capability(Mode.NATIVE, Spark.V2, "PII")

BigTable Source

@Capability(Mode.CLOUD, "PII")

AWS S3 Source

@Capability(Mode.CLOUD, Mode.NATIVE)

mode_cloud, PII

Filtered out

Visible

Filtered out

Preview

A pipeline preview runs in native (local machine) JVM. Hence, any plugin which is capable of running in Native environment is capable of preview. If a plugin which is not capable of running in Native is run in a preview then that preview operation will fail with incapability as soon as the preview is started and before the actual preview process runs.

Failing Early and Gracefully

Filtering incapable plugins does not guarantee that a pipeline containing an incapable plugin will not be run in the system. This might happen because of various reason for example, a pipeline was created before a capability was required i.e. the capability was added as a requirement in the plugin.required.capabilities configuration after the pipeline deployment.

Whenever an incapable plugin is encountered the pipeline should fail early and gracefully. Pipeline can at different stages: 

1. Deploy time: Since plugins which are incapable to run will be filtered out, a user will not be able to create a pipeline using them. If a user imports a pipeline json which contains an incapable plugin then the deployment will fail with Artifact/Plugin not found exception. Note: The exception here will not be Incapability exception since that will expose the existence of the plugin in the system which is not ideal.
2. Run Time (Before run): In addition to filtering a plugin we checks before provisioning will also be performed. This will fail the pipeline if any of the plugin is not capable to run in the provisioned environment. For example running a plugin which AWS which is not capable to run in AWS and can only run in Azure.

When a pipeline is failed due to incapability the failure will be surfaced to user through logs and also an error message in UI.

Plugin changes

The above plugin design tries to minimize the required plugin changes. If a plugin does not specify any capability option then the default capability will be used according to which the plugin will be considered as capable of all system defined capabilities but not with any user defined capability options such as PII. Custom capabilities needs to be specifically defined by the plugin developer. The system defined capabilities with which a plugin will be considered capable in absence of any specified capability is following:

1. Native
2. Remote
3. Spark 1
4. Spark 2

No plugin change is required after CDAP upgrade to 5.1 as plugins from previous version will be considered capable with all system defined capability as mentioned above.

Although to filter out all the plugins which require transactions or is known to be compatible with only one version of Spark will need to be annotated so that the default capability is not inferred for them. The below table list all the plugins which will need to be specifically annotated and the annotation required: