...

New CDAP Features

App Verification

After 3.2, a user can deploy an artifact, then create an app by specifying an artifact and a configuration.  At configure time, an app can perform some validation on its config and throw an exception to fail app creation.  Sometimes, it would be useful just to run the configure() method without actually creating any streams, datasets, or app, in order to let the app validate its config.  For example, in Hydrator, there is a validate button, which is implemented completely in the UI.  The UI really shouldn't be validating anything, as all that logic is specific to the application code.

We could let the application throw a specific exception for cases where its config is invalid

public class InvalidConfigurationException extends RuntimeException {
  public InvalidConfigurationException(String message) {
    super(message);
  }
}
 
public interface Application<T extends Config> {
  /**
   * Configures the Application.
   *
   * @param configurer Collects the Application configuration
   * @param context Used to access the environment, application configuration, and application (deployment) arguments
   * @throws InvalidConfigurationException if the config is invalid
   */
  void configure(ApplicationConfigurer configurer, ApplicationContext<T> context);
}

If thrown, app fabric will return a 400 with the message of the exception. We can also add a dryrun option to the create app request that will call configure() without creating anything. In the end, it returns the ApplicationSpecification, which is sent as the output of the request.  

PUT /apps/{app-name} -H 'Content-Type: application/json' -d '{
  "artifact": { ... },
  "config": { ... },
  "dryrun": true
}'

New Plugins

Kinesis source:

• User wants to read from Kinesis source (AWS) and persist it into a dataset, as data in AWS kinesis stream is available only for 24hrs.
  
  

  Code Block
  "config": { "source": { "name": "MyKinesisSource", "plugin": { "name": "Kinesis", "properties": { "credentials": "...", "shardId" : "id442222", "limit" : 5MBps // we can skip this if we use KCL rather than the REST API (have to investigate) } } .... } }

...