Table of Contents |
---|
Checklist
- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
Introduction
CDAP pipeline is composed of various plugins that can be configured by users as CDAP pipelines are being developed. While building CDAP pipelines, pipeline developer can provide invalid plugin configurations or schema. For example, the BigQuery sink plugin can have output schema which does not match with underlying BigQuery table. CDAP pipeline developer can use new validation endpoint to validate the stages before deploying the pipeline. In order to fail fast and for better user experience, validation endpoint should return all the validation errors from a given stage when this endpoint is called.
Data pipeline app exposes various error types for plugin validation. In future releases, new error types can be introduced. With current implementation, when plugins with new error types are pushed to hub, data pipeline artifacts need to be updated for every new type of error that is introduced. This is because the validation errors are defined in the data pipeline app itself. A better approach would be to modify data pipeline app so that app artifacts do not need to be replaced for every new type of error.
Goals
To fail fast and for better user experience, introduce a new api to collect multiple validation error messages from a stage at configure time
- Decouple validation error types from data pipeline app
Instrument plugins to use this api to return multiple error messages for validation endpoint
User Stories
- As a CDAP pipeline developer, when I validate a stage, I expect that all the invalid config properties and input/output schema fields are highlighted on CDAP UI with appropriate error message and corrective action.
- As a plugin developer, I should be able to capture all the validation errors while configuring the plugin so that all the validation errors can be surfaced on CDAP UI.
- As a plugin developer, I should be able to use new validation error types without replacing data pipeline app artifacts.
API Changes for Plugin Validation
Collect Multiple errors from plugins
To collect multiple stage validation errors from the stage, StageConfigurer, MultiInputStageConfigurer and MultiOutputStageConfigurer can be modified as below. Current implementation does not expose stage name to the plugin in configurePipeline method. Stage name will be needed by the plugins to create stage specific errors. For that, stage name will be exposed to plugins through stage configurer as below.
Code Block | ||||
---|---|---|---|---|
| ||||
public interface StageConfigurer { ... /** * getGet the stage name. * * @return stage name */ String getStageName(); |
Decouple plugin error types from data pipeline app
Approach - 1
To carry error information, a new ValidationFailure class is introduced to collect multiple validation failures in stage configurer. This class can be built using a ValidationFailureBuilder which only allows string properties. The builder expose methods to get message, type and properties of a failure. The validation failures are collected using ValidationException. Using this validation exception whenever plugin has an invalid property that is tied to another invalid property, plugin can throw a validation exception with all the errors collected so far. This keep plugin validation code much simpler.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Validation failure. */ @Beta public class ValidationFailure { /** * Adds a new validation failure to the configurer. * * @param failure a validation failure */ void addValidationFailure(ValidationFailure failure); /** * Throws validation exception if there are any failures that are added to the configurer through * addValidationFailure method. * * @throws ValidationException if there are any validation failures being carried by the configurer */ void throwIfFailure() throws ValidationException; |
Decouple plugin error types from data pipeline app
Approach - 1
To carry error information, a new ValidationFailure class is introduced to collect multiple validation failures in stage configurer. This class can be built using a ValidationFailureBuilder which only allows string properties. The builder expose methods to get message, type and properties of a failure. The validation failures are collected using ValidationException. Using this validation exception whenever plugin has an invalid property that is tied to another invalid property, plugin can throw a validation exception with all the errors collected so far. This keep plugin validation code much simpler.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents an error condition occurred during validation. */ @Beta public class ValidationFailure { // types of the failures private static final String STAGE_ERROR = "StageError"; private static final String INVALID_PROPERTY = "InvalidProperty"; private static final String PLUGIN_NOT_FOUND = "PluginNotFound"; private static final String INVALID_INPUT_SCHEMA = "InvalidInputSchema"; private static final String INVALID_OUTPUT_SCHEMA = "InvalidOutputSchema"; // represents stage name in the failure. It is a generic property used in all the failures for a given stage private static final String STAGE = "stage"; // represents configuration property in InvalidProperty failure private static final String CONFIG_PROPERTY = "configProperty"; // represents plugin id in PluginNotFound failure private static final String PLUGIN_ID = "pluginId"; // represents plugin type in PluginNotFound failure private static final String PLUGIN_TYPE = "pluginType"; // represents plugin name in PluginNotFound failure private static final String PLUGIN_NAME = "pluginName"; // represents a field in InvalidInputSchema or InvalidOutputSchema failure private static final String FIELD = "field"; // represents input stage in InvalidInputSchema failure private static final String INPUT_STAGE = "inputStage"; // represents output port in InvalidOutputSchema failure private static final String OUTPUT_PORT = "outputPort"; private final String message; private final String type; private final String correctiveAction; private final Map<String, Object> properties; private ValidationFailure(String message, String type, @Nullable String correctiveAction, Map<String, Object> properties) { this.message = message; this.type = type; this.correctiveAction = correctiveAction; this.properties = properties; } /** * Creates a stage validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createStageFailure(String message, String stage, @Nullable String correctiveAction) { Builder builder = builder(message, STAGE_ERROR); builder.setCorrectiveAction(correctiveAction).addProperty(STAGE, stage); return builder.build(); } /** * Creates a config property validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createConfigPropertyFailure(String message, String stage, String property, @Nullable String correctiveAction) { Builder builder = builder(message, INVALID_PROPERTY); builder.setCorrectiveAction(correctiveAction) .addProperty(STAGE, stage).addProperty(CONFIG_PROPERTY, property); return builder.build(); } /** * Creates a plugin not found validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createPluginNotFoundFailure(String message, String stage, String pluginId, String pluginName, String pluginType, @Nullable String correctiveAction) { Builder builder = builder(message, PLUGIN_NOT_FOUND); builder.setCorrectiveAction(correctiveAction) .addProperty(STAGE, stage).addProperty(PLUGIN_ID, pluginId) .addProperty(PLUGIN_TYPE, pluginType) .addProperty(PLUGIN_NAME, pluginName); return builder.build(); } /** * Creates a invalid input schema failure. * * @param message validation failure message * @param stage stage name * @param field input schema field * @param inputStage optional input stagename. This is applicable to plugins of type {@link Joiner}. * @param correctiveAction optional corrective action * @return invalid input schema validation failure */ public static ValidationFailure createInputSchemaFailure(String message, String stage, String field, @Nullable String inputStage, @Nullable String correctiveAction) { ... } /** * Creates a invalid output schema failure. * * @param message validation failure message * @param stage stage name * @param field output schema field * @param outputPort optional output port. This is applicable to plugins of type {@link SplitterTransform}. * @param correctiveAction optional corrective action * @return invalid output schema validation failure */ public static ValidationFailure createOutputSchemaFailure(String message, String stage, String field, @Nullable String outputPort, @Nullable String correctiveAction) { .... } /** * Returns a builder for creating a {@link ValidationFailure}. */ public static Builder builder(String message, String type) { return new Builder(message, type); } /** * A builder to create {@link ValidationFailure} instance. */ public static class Builder { private final String message; private final String type; private final String correctiveAction; private final Map<String, Object> properties; ValidationFailure(String message, String type, String correctiveAction, Map<String, Object> mapprivate Builder(String message, String type) { this.message = message; this.type = type; this.correctiveActionproperties = new correctiveActionHashMap<>(); } this.properties = map; /** } * Sets corrective publicaction staticto ValidationFailure.Builder builder(String message, String type, String correctiveAction) { rectify the failure. * return new ValidationFailure.Builder(message, type, correctiveAction); } * @param correctiveAction corrective action public* static@return classthis Builderbuilder { private*/ final String message; public Builder setCorrectiveAction(String privatecorrectiveAction) final{ String type; this.correctiveAction = correctiveAction; private final Map<String, Object> propertiesreturn this; } private Builder(String message, String type, String/** correctiveAction) { * Adds a this.messageproperty =to message;the failure. this.type* = type; * @param property this.correctiveActionthe =name correctiveAction;of the property this.properties =* new HashMap<>(); } @param value the value of the property // methods to* get@return variousthis typesbuilder of properties such as string, schema.. */ public ValidationFailure.Builder addProperty(String property, String value) { this.properties.put(property, value); return this; } ..../** public ValidationFailure* build()Creates {a new instance of {@link ValidationFailure}. return new ValidationFailure(message, type, correctiveAction,* properties); }* @return instance } } | ||||
Code Block | ||||
| ||||
/** * Represents Validation Exception.of {@link ValidationFailure} */ @Beta public class ValidationExceptionpublic extendsValidationFailure RuntimeExceptionbuild() { private List<ValidationFailure> failures; return new private ValidationException(List<ValidationFailure> failures) {ValidationFailure(message, type, correctiveAction, super(); this.failures = failures; } @Override public String getMessage() { ... } Collections.unmodifiableMap(new HashMap<>(properties))); /** } * Returns list of} failures. */ public List<ValidationFailure> getFailures() { return failures; } /** * Returns message of this failure. */ public static ValidationException.Builder builderString getMessage() { return new ValidationException.Builder()message; } /** * BuilderReturns totype buildof validationthis exceptionfailure. */ public static class BuilderString getType() { private final List<ValidationFailure> failuresreturn type; } private Builder() { /** this.failures = new ArrayList<>(); } public ValidationException.Builder addFailure(ValidationFailure failure) { failures.add(failure); * Returns corrective action for this failure. */ @Nullable public String getCorrectiveAction() { return thiscorrectiveAction; } /** * UtilReturns methodproperties toof createthis andfailure. add stage validation failure*/ to this exception.public Map<String, Object> getProperties() { * return properties; * @param} message validation failure message@Override public boolean equals(Object *o) @param{ stage stage name if (this == *o) @param{ correctiveAction suggested action return true; */ } public ValidationException.Builder addStageValidationFailure(String message, Stringif stage,(o == null || getClass() != o.getClass()) { return false; } ValidationFailure that = (ValidationFailure) o; return message.equals(that.message) && type.equals(that.type) && Objects.equals(correctiveAction, that.correctiveAction) && properties.equals(that.properties); } @Override public @Nullable String correctiveActionint hashCode() { return ValidationFailure.Builder builder = ValidationFailure.builderObjects.hash(message, "INVALID_STAGE"type, correctiveAction, properties); } builder.addProperty("stage", stage);@Override public failures.add(builder.build()); String toString() { return this; } "ValidationFailure{" + // util methods for other validation failures ..."message='" + message + '\'' + /** * Build and throw validation exception. This method can be used by plugins to throw an exception at any point * during validation. The method will build and throw ValidationException if there are any failures added while * building the exception * * @return Validation exception */ public void throwIfAbsent() { if (failures.isEmpty()) { return; } throw new ValidationException(failures); }", type='" + type + '\'' + ", correctiveAction='" + correctiveAction + '\'' + ", properties=" + properties + '}'; } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/**
* Validation exception that carries multiple validation failures.
*/
@Beta
public class ValidationException extends RuntimeException {
private List<ValidationFailure> failures;
public ValidationException(List<ValidationFailure> failures) {
super(failures.isEmpty() ? "Validation Exception occurred." : failures.iterator().next().getMessage());
this.failures = failures;
}
/**
* Returns list of failures.
*/
public List<ValidationFailure> getFailures() {
return failures;
}
} |
API usage in plugins
Code Block |
---|
@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class); StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer(); // get the name of the stage String stageName = stageConfigurer.getStageName(); ValidationException.Builder exceptionBuilder = ValidationException.builder(); try { Pattern.compile(conf.filterRegex); } catch (Exception e) { // add validation failure to stage configurer exceptionBuilder.addStageValidationFailurestageConfigurer.addValidationFailure(ValidationFailure.createConfigPropertyFailure(e.getMessage(), stageName, "filterRegex", "Make sure the file regex is correct")); } if (conf.sourceFileset.equals(conf.destinationFileset)) { // add validation failure to stage configurer exceptionBuilderstageConfigurer.addValidationFailure(ValidationFailure.createStageFailure("source and destination filesets must be different", stageName, "Provide different source and destination filesets")); } exceptionBuilderstageConfigurer.throwIfAbsentthrowIfFailure(); } |
Approach - 2
Validation error represents an error with various causes with different attributes for each cause. For example, when the input schema field type does not match the underlying sink schema, the cause is input field mismatch with attributes such as stage name, field name, suggested type etc. Each error message can be associated to more than one causes. This can happen for plugins such as joiner and splitter where there are multiple input or output schemas from a given stage. For example, when input schemas for joiner are not compatible, the causes will include mismatching fields from input schemas of incoming stages. This means that a validation error can be represented as a list of causes where each cause is a map of cause attribute to its value as shown below.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents failure that occurred during validation. */ @Beta public class ValidationFailure { private final String message; protected final List<Map<String, Object>> causes; /** * Creates a validation failure with a message and empty map of causes * @param message */ public ValidationFailure(String message) { this.message = message; this.causes = new ArrayList<>(); } @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } ValidationFailure that = (ValidationFailure) o; return message.equals(that.message) && causes.equals(that.causes); } @Override public int hashCode() { return Objects.hash(message, causes); } } |
All the attributes of a cause can be tracked at central location as below:
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Failure attributes. */ public enum FailureAttributes { STAGE("stage"), // represents stage being validated PROPERTY("property"), // represents stage property INPUT_FIELD("inputField") // represents field in the input schema OUTPUT_FIELD("outputField") // represents field in the output schema OUTPUT_PORT("outputPort"), // represents output port for plugins such as SplitterTransform where multiple output schemas are expected INPUT_STAGE("inputStage"), // represents input stage for plugins such as Joiner where multiple input schemas are expected .. private String name; FailureAttributes(String name) { this.name = name; } } |
Introduced Errors
With this approach following error classes can be added to hydrator-common which represents specific type of errors.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents failure that occurred during stage validation. */ @Beta public class InvalidStageFailure extends ValidationFailure { /** * Creates validation failure that occurred during stage validation. * @param message failure message * @param stage name of the stage that caused this validation failure */ public InvalidStageFailure(String message, String stage) { super(message); causes.add(Collections.singletonMap("stage", stage)); } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents failure that occurred during stage config property validation. */ @Beta public class InvalidStagePropertyFailure extends ValidationFailure { /** * Creates validation failure that occurred during stage validation. * @param message failure message * @param stage name of the stage that caused this validation failure * @param property property that is invalid */ public InvalidStageFailure(String message, String stage, String property) { super(message); Map<String, Object> map = new HashMap<>(); map.put("stage", stage); map.put("property", property); causes.add(map); } /** * Creates validation failure that occurred during stage validation. * @param message failure message * @param stage name of the stage that caused this validation failure * @param properties properties that is caused this failure */ public InvalidStageFailure(String message, String stage, String[] properties) { super(message); Map<String, Object> map = new HashMap<>(); for (String property : properties) { map.put("stage", stage); map.put("property", property); } causes.add(map); } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents invalid input schema failure. */ public class InvalidInputSchemaFailure extends ValidationFailure { /** * Creates invalid input schema failure. * @param message failure message * @param stage name of the stage * @param map map of incoming stage name to field that is invalid. */ public InvalidInputSchemaFailure(String message, String stage, Map<String, String> map) { super(message); for (Map.Entry<String, String> entry : map.entrySet()) { Map<String, Object> causeMap = new HashMap<>(); causeMap.put("stage", stage); causeMap.put("inputStage", entry.getKey()); causeMap.put("inputField", entry.getValue()); causes.add(causeMap); } } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents invalid output schema failure. */ public class InvalidOutputSchemaFailure extends ValidationFailure { /** * Creates invalid output schema failure. * @param message failure message * @param stage name of the stage * @param map map of output going port name to field that is invalid */ public InvalidOutputSchemaFailure(String message, String stage, Map<String, String> map) { super(message); for (Map.Entry<String, String> entry : map.entrySet()) { Map<String, Object> causeMap = new HashMap<>(); causeMap.put("stage", stage); causeMap.put("outputPort", entry.getKey()); causeMap.put("outputField", entry.getValue()); causes.add(causeMap); } } } |
API usage in plugins
Code Block |
---|
@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class); StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer(); // get the name of the stage String stageName = stageConfigurer.getStageName(); try { Pattern.compile(conf.filterRegex); } catch (Exception e) { // add validation error to stage configurer stageConfigurer.addValidationFailure(new InvalidStagePropertyFailure(e.getMessage(), stageName, "filterRegex")); } if (conf.sourceFileset.equals(conf.destinationFileset)) { // add validation error to stage configurer stageConfigurer.addValidationFailure(new InvalidStageFailure("source and destination filesets must be different", stageName)); } } |
Impact on UI
Type | Description | Scenario | Approach - 1 - Json Response | Approach - 2 - Json Response |
---|---|---|---|---|
StageError | Represents validation error while configuring the stage | If there is any error while connecting to sink while getting actual schema | { | { "correctiveAction" : "Make sure correct driver is uploaded.", |
InvalidProperty | Represents invalid configuration property | If config property value contains characters that are not allowed by underlying source or sink | { | { "correctiveAction" : "Either drop or keep should be empty", |
PluginNotFound | Represents plugin not found error for a stage. This error will be added by the data pipeline app | If the plugin was not found. This error will be thrown from the data pipeline app | { | { "correctiveAction" : "Please make sure the 'Mock' plugin is installed.", "pluginId" : "Mock" |
InvalidInputSchema | Represents invalid schema field in input schema | If the input schemas for joiner plugin is of different types | { { | { { ] |
InvalidOutputSchema | Represents invalid schema field in output schema | If the output schema for the plugin is not compatible with underlying sink | { | { "correctiveAction" : "Schema should be of type 'string' at output port 'port'", |
Conclusion
There are 2 contracts in this design. Programmatic contract between data pipeline app and plugins and another between data pipeline app and UI. Approach 1 provides well defined programmatic contract between plugins and data pipeline app. Using programmatic apis of Approach 1, its possible to throw the exception with all the collected errors at any point from plugin validation code in case one of the dependent properties are invalid. This makes the plugin validation code much more simpler. Hence, Approach 1 is suggested.
Related Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|