Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Checklist

  •  User Stories Documented
  •  User Stories Reviewed
  •  Design Reviewed
  •  APIs reviewed
  •  Release priorities assigned
  •  Test cases reviewed
  •  Blog post

Introduction 

CDAP pipeline is composed of various plugins that can be configured by users as CDAP pipelines are being developed. While building CDAP pipelines, pipeline developer can provide invalid plugin configurations. For example, the BigQuery sink plugin can have schema which does not match with an underlying BigQuery table. CDAP pipeline developer can use new validation endpoint to validate the stages before deploying the pipeline. In order to fail fast, validation endpoint should return all the validation errors from a given stage when this endpoint is called. 

Data pipeline app exposes various error types for plugin validation. In future releases, new error types can be introduced. When plugins with new error types are pushed to hub, data pipeline artifacts need to be upgraded for every new type of error that is introduced. This is because the validation errors are defined in the data pipeline app itself. A better approach would be to modify data pipeline app so that app artifacts do not need to be replaced for every new type of error.

Goals

  • To fail fast, introduce a new api to collect multiple error messages from plugins at configure time

  • Decouple various validation error types from data pipeline app
  • Instrument plugins to use this api to return multiple error messages for validation endpoint

User Stories 

  • As a CDAP pipeline developer, when I validate a stage, I expect that all the invalid config properties and schema fields are highlighted on CDAP UI with appropriate error message.
  • As a plugin developer, I should be able to specify all the validation errors while configuring the plugin for better user experience.
  • As a plugin developer, I should be able to use new validation error types without replacing data pipeline app artifacts. 

API Changes for Plugin Validation

Collect multiple errors from plugins 

To collect multiple stage validation errors from the stage, StageConfigurer, MultiInputStageConfigurer and MultiOutputStageConfigurer can be modified as below. If there are any validation errors added to stage configurer, the pipeline deployment will fail and all the errors will be returned as a response to stage validation REST endpoint.

Current implementation does not expose stage name to the plugin in configurePipeline method. Stage name will be needed by the plugins to create stage specific errors. For that stageName will be exposed to through stage configurer as below.

Code Block
languagejava
titleStageConfigurer.java
public interface StageConfigurer {

  ...

/**
 * get the stage name.
 * @return stage name
 */
String getStageName();

/**
 * add validation errors.
 * @param e errors
 */
void addValidationError(ValidationError e);


Decoupling plugin error types from data pipeline app

A new ValidationError class is introduced to collect multiple validation errors in stage configurer. This class will be exposed by data pipeline app to the plugins. Each new type of error can be added to hydrator-common which can be added as compile time dependency to the plugins. This approach allows us to easily add more types of validation errors for plugins while removing a need to update data pipeline artifact for each new error type. Below are some of the error types that will be added to hydrator-common module.

Code Block
languagejava
titleValidationError.java
/**
 * Represents error that occurred during validation.
 */
@Beta
public class ValidationError {
  private final String message;
  private final String type;

  /**
   * Creates an error with provided error message.
   * @param message error message
   */
  public ValidationError(String message) {
    this.message = message;
    this.type = getType();
  }

  /**
   * Returns the type of the error.
   */
  public String getType() {
    return "ERROR";
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    ValidationError error = (ValidationError) o;
    return message.equals(error.message) &&
      type.equals(error.type);
  }

  @Override
  public int hashCode() {
    return Objects.hash(message, type);
  }
}

Introduced Errors

Following errors will be added to hydrator-common. Note that this list will keep evolving as new types of errors are added to the module.

Code Block
languagejava
titleInvalidStageError.java
public class InvalidStageError extends ValidationError {
  private final String stage;

  /**
   * Creates error that represents a stage validation error.
   *
   * @param message error message
   * @param stage name of the stage
   */
  public InvalidStageError(String message, String stage) {
    super(message);
    this.stage = stage;
  }

  @Override
  public String getType() {
    return "STAGE_ERROR";
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    if (!super.equals(o)) {
      return false;
    }
    InvalidStageError that = (InvalidStageError) o;
    return Objects.equals(stage, that.stage);
  }

  @Override
  public int hashCode() {
    return Objects.hash(super.hashCode(), stage);
  }
}
Code Block
languagejava
titleInvalidStagePropertyError.java
/**
 * Represents invalid stage property error that occurred during stage validation.
 */
@Beta
public class InvalidStagePropertyError extends InvalidStageError {
  private final String property;

  /**
   * Creates error that represents invalid stage property.
   *
   * @param message error message
   * @param stage name of the stage
   * @param property invalid property
   */
  public InvalidStagePropertyError(String message, String stage, String property) {
    super(message, stage);
    this.property = property;
  }

  @Override
  public String getType() {
    return "INVALID_FIELD";
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    if (!super.equals(o)) {
      return false;
    }
    InvalidStagePropertyError that = (InvalidStagePropertyError) o;
    return property.equals(that.property);
  }

  @Override
  public int hashCode() {
    return Objects.hash(super.hashCode(), property);
  }
}
Code Block
languagejava
titleInvalidSchemaFieldError.java
/**
 * Represents schema mismatch that occurred during stage validation.
 */
@Beta
public class InvalidSchemaFieldError extends InvalidStageError {
  private final String field;
  private final boolean isInputField;

  /**
   * Creates error that represents schema mismatch
   *
   * @param message error message
   * @param stage name of the stage
   * @param field invalid field
   * @param isInputField true indicates that the field is from input schema.
   *                     false indicates that the field is from output schema
   */
  public InvalidSchemaFieldError(String message, String stage, String field, boolean isInputField) {
    super(message, stage);
    this.field = field;
    this.isInputField = isInputField;
  }

  @Override
  public String getType() {
    return "INVALID_SCHEMA";
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    if (!super.equals(o)) {
      return false;
    }
    InvalidSchemaFieldError that = (InvalidSchemaFieldError) o;
    return isInputField == that.isInputField && field.equals(that.field);
  }

  @Override
  public int hashCode() {
    return Objects.hash(super.hashCode(), field, isInputField);
  }
}

API usage in plugins

Code Block
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
  pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class);
  StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
  // get the name of the stage 
  String stageName = stageConfigurer.getStageName();
  try {
    Pattern.compile(conf.filterRegex);
  } catch (Exception e) {  
    // add validation error to stage configurer
    stageConfigurer.addValidationError(new InvalidStagePropertyError(e.getMessage(), stageName, "filterRegex"));
  }
  if (conf.sourceFileset.equals(conf.destinationFileset)) {
    // add validation error to stage configurer
    stageConfigurer.addValidationError(new InvalidStageError("source and destination filesets must be different", stageName));
  }
}

Impact on UI

UI should be able to handle new error types that are introduced. For example, for invalid stage properties, UI should highlight all the invalid properties for a given stage. For schema mismatch, UI should be able to highlight schema fields that are mismatching. 

Below are the responses to the validation endpoint for each type of validation error:

TypeDescriptionJson Response
STAGE_ERRORRepresents validation error while configuring the stage

{
  "type" "STAGE_ERROR",
  "stage" "src",
  "message" : "source and destination filesets must be different"
}

INVALID_FIELDRepresents invalid configuration property
{
  "type" : "INVALID_FIELD", 
  "stage" : "src",
  "message" : "Invalid config for property 'port'", 
"property" : "port"
}
PLUGIN_NOT_FOUNDRepresents plugin not found error for a stage
{
"errors": [
{
"stage": "src",
"type": "PLUGIN_NOT_FOUND",
"message": "Plugin named 'Mock' of type 'batchsource' not found.",
"pluginType": "batchsource",
"pluginName": "Mock",
"requestedArtifact": {
"scope": "USER",
"name": "app-mocks-ghost",
"version": "1.0.0"
},
"suggestedArtifact": {
"scope": "USER",
"name": "app-mocks",
"version": "1.0.0"
}
}
]
}
INVALID_SCHEMARepresents invalid schema field
{
  "type" : "INVALID_SCHEMA", 
  "stage" : "sink",
  "message" : "Invalid schema for the field 'name'", 
"field" : "name",
  "isInputField" : false
}

Related Jira

Jira Legacy
serverCask Community Issue Tracker
serverId45b48dee-c8d6-34f0-9990-e6367dc2fe4b
keyCDAP-15578

Future Work

  • Add suggestions about how the error can be corrected
  • Add pipeline level validation such as DAG validation

Test Scenarios

Test IDTest DescriptionExpected Results












Releases

Release 6.1.0

Related Work