Table of Contents |
---|
Checklist
- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
Introduction
CDAP pipeline is composed of various plugins that can be configured by users as CDAP pipelines are being developed. While building CDAP pipelines, pipeline developer can provide invalid plugin configurations or schema. For example, the BigQuery sink plugin can have output schema which does not match with underlying BigQuery table. CDAP pipeline developer can use new validation endpoint to validate the stages before deploying the pipeline. In order to fail fast and for better user experience, validation endpoint should return all the validation errors from a given stage when this endpoint is called.
Data pipeline app exposes various error types for plugin validation. In future releases, new error types can be introduced. With current implementation, when plugins with new error types are pushed to hub, data pipeline artifacts need to be updated for every new type of error that is introduced. This is because the validation errors are defined in the data pipeline app itself. A better approach would be to modify data pipeline app so that app artifacts do not need to be replaced for every new type of error.
Goals
To fail fast and for better user experience, introduce a new api to collect multiple validation error messages from a stage at configure time
- Decouple validation error types from data pipeline app
Instrument plugins to use this api to return multiple error messages for validation endpoint
User Stories
- As a CDAP pipeline developer, when I validate a stage, I expect that all the invalid config properties and input/output schema fields are highlighted on CDAP UI with appropriate error message and corrective action.
- As a plugin developer, I should be able to capture all the validation errors while configuring the plugin so that all the validation errors can be surfaced on CDAP UI.
- As a plugin developer, I should be able to use new validation error types without replacing data pipeline app artifacts.
API Changes for Plugin Validation
Collect Multiple errors from plugins
To collect multiple stage validation errors from the stage, StageConfigurer, MultiInputStageConfigurer and MultiOutputStageConfigurer can be modified as below. Current implementation does not expose stage name to the plugin in configurePipeline method. Stage name will be needed by the plugins to create stage specific errors. For that, stage name will be exposed to plugins through stage configurer as below.
Code Block | ||||
---|---|---|---|---|
| ||||
public interface StageConfigurer { ... /** * Get the stage name. * * @return stage name */ String getStageName(); /** * Adds a new validation failure to the configurer. * * @param failure a validation failure */ void addValidationFailure(ValidationFailure failure); /** * Throws validation exception if there are any failures that are added to the configurer through * addValidationFailure method. * * @throws ValidationException if there are any validation failures being carried by the configurer */ void throwIfFailure() throws ValidationException; |
Decouple plugin error types from data pipeline app
Approach - 1
To carry error information, a new ValidationFailure class is introduced to collect multiple validation failures in stage configurer. This class can be built using a ValidationFailureBuilder which only allows string properties. The builder expose methods to get message, type and properties of a failure. The validation failures are collected using ValidationException. Using this validation exception whenever plugin has an invalid property that is tied to another invalid property, plugin can throw a validation exception with all the errors collected so far. This keep plugin validation code much simpler.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents an error condition occurred during validation. */ @Beta public class ValidationFailure { // types of the failures private static final String STAGE_ERROR = "StageError"; private static final String INVALID_PROPERTY = "InvalidProperty"; private static final String PLUGIN_NOT_FOUND = "PluginNotFound"; private static final String INVALID_INPUT_SCHEMA = "InvalidInputSchema"; private static final String INVALID_OUTPUT_SCHEMA = "InvalidOutputSchema"; // represents stage name in the failure. It is a generic property used in all the failures for a given stage private static final String STAGE = "stage"; // represents configuration property in InvalidProperty failure private static final String CONFIG_PROPERTY = "configProperty"; // represents plugin id in PluginNotFound failure private static final String PLUGIN_ID = "pluginId"; // represents plugin type in PluginNotFound failure private static final String PLUGIN_TYPE = "pluginType"; // represents plugin name in PluginNotFound failure private static final String PLUGIN_NAME = "pluginName"; // represents a field in InvalidInputSchema or InvalidOutputSchema failure private static final String FIELD = "field"; // represents input stage in InvalidInputSchema failure private static final String INPUT_STAGE = "inputStage"; // represents output port in InvalidOutputSchema failure private static final String OUTPUT_PORT = "outputPort"; private final String message; private final String type; private final String correctiveAction; private final Map<String, Object> properties; private ValidationFailure(String message, String type, @Nullable String correctiveAction, Map<String, Object> properties) { this.message = message; this.type = type; this.correctiveAction = correctiveAction; this.properties = properties; } /** * Creates a stage validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createStageFailure(String message, String stage, @Nullable String correctiveAction) { Builder builder = builder(message, STAGE_ERROR); builder.setCorrectiveAction(correctiveAction).addProperty(STAGE, stage); return builder.build(); } /** * Creates a config property validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createConfigPropertyFailure(String message, String stage, String property, @Nullable String correctiveAction) { Builder builder = builder(message, INVALID_PROPERTY); builder.setCorrectiveAction(correctiveAction) .addProperty(STAGE, stage).addProperty(CONFIG_PROPERTY, property); return builder.build(); } /** * Creates a plugin not found validation failure. * * @param message validation failure message * @param stage stage name * @param correctiveAction corrective action */ public static ValidationFailure createPluginNotFoundFailure(String message, String stage, String pluginId, String pluginName, String pluginType, @Nullable String correctiveAction) { Builder builder = builder(message, PLUGIN_NOT_FOUND); builder.setCorrectiveAction(correctiveAction) .addProperty(STAGE, stage).addProperty(PLUGIN_ID, pluginId) .addProperty(PLUGIN_TYPE, pluginType) .addProperty(PLUGIN_NAME, pluginName); return builder.build(); } /** * Creates a invalid input schema failure. * * @param message validation failure message * @param stage stage name * @param field input schema field * @param inputStage optional input stagename. This is applicable to plugins of type {@link Joiner}. * @param correctiveAction optional corrective action * @return invalid input schema validation failure */ public static ValidationFailure createInputSchemaFailure(String message, String stage, String field, @Nullable String inputStage, @Nullable String correctiveAction) { ... } /** * Creates a invalid output schema failure. * * @param message validation failure message * @param stage stage name * @param field output schema field * @param outputPort optional output port. This is applicable to plugins of type {@link SplitterTransform}. * @param correctiveAction optional corrective action * @return invalid output schema validation failure */ public static ValidationFailure createOutputSchemaFailure(String message, String stage, String field, @Nullable String outputPort, @Nullable String correctiveAction) { .... } /** * Returns a builder for creating a {@link ValidationFailure}. */ public static Builder builder(String message, String type) { return new Builder(message, type); } /** * A builder to create {@link ValidationFailure} instance. */ public static class Builder { private final String message; private final String type; private String correctiveAction; private final Map<String, Object> properties; private Builder(String message, String type) { this.message = message; this.type = type; this.properties = new HashMap<>(); } /** * Sets corrective action to rectify the failure. * * @param correctiveAction corrective action * @return this builder */ public Builder setCorrectiveAction(String correctiveAction) { this.correctiveAction = correctiveAction; return this; } /** * Adds a property to the failure. * * @param property the name of the property * @param value the value of the property * @return this builder */ public Builder addProperty(String property, String value) { this.properties.put(property, value); return this; } /** * Creates a new instance of {@link ValidationFailure}. * * @return instance of {@link ValidationFailure} */ public ValidationFailure build() { return new ValidationFailure(message, type, correctiveAction, Collections.unmodifiableMap(new HashMap<>(properties))); } } /** * Returns message of this failure. */ public String getMessage() { return message; } /** * Returns type of this failure. */ public String getType() { return type; } /** * Returns corrective action for this failure. */ @Nullable public String getCorrectiveAction() { return correctiveAction; } /** * Returns properties of this failure. */ public Map<String, Object> getProperties() { return properties; } @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } ValidationFailure that = (ValidationFailure) o; return message.equals(that.message) && type.equals(that.type) && Objects.equals(correctiveAction, that.correctiveAction) && properties.equals(that.properties); } @Override public int hashCode() { return Objects.hash(message, type, correctiveAction, properties); } @Override public String toString() { return "ValidationFailure{" + "message='" + message + '\'' + ", type='" + type + '\'' + ", correctiveAction='" + correctiveAction + '\'' + ", properties=" + properties + '}'; } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Validation exception that carries multiple validation failures. */ @Beta public class ValidationException extends RuntimeException { private List<ValidationFailure> failures; public ValidationException(List<ValidationFailure> failures) { super(failures.isEmpty() ? "Validation Exception occurred." : failures.iterator().next().getMessage()); this.failures = failures; } /** * Returns list of failures. */ public List<ValidationFailure> getFailures() { return failures; } } |
API usage in plugins
Code Block |
---|
@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class); StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer(); // get the name of the stage String stageName = stageConfigurer.getStageName(); try { Pattern.compile(conf.filterRegex); } catch (Exception e) { // add validation failure to stage configurer stageConfigurer.addValidationFailure(ValidationFailure.createConfigPropertyFailure(e.getMessage(), stageName, "filterRegex", "Make sure the file regex is correct")); // plugin can choose to terminate processing here stageConfigurer.throwIfFailure(); } if (conf.sourceFileset.equals(conf.destinationFileset)) { // add validation failure to stage configurer stageConfigurer.addValidationFailure(ValidationFailure.createStageFailure("source and destination filesets must be different", stageName, "Provide different source and destination filesets")); } } |
Approach - 2
Validation error represents an error with various causes with different attributes for each cause. For example, when the input schema field type does not match the underlying sink schema, the cause is input field mismatch with attributes such as stage name, field name, suggested type etc. Each error message can be associated to more than one causes. This can happen for plugins such as joiner and splitter where there are multiple input or output schemas from a given stage. For example, when input schemas for joiner are not compatible, the causes will include mismatching fields from input schemas of incoming stages. This means that a validation error can be represented as a list of causes where each cause is a map of cause attribute to its value as shown below.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * AddFailure collector validationis failureresponsible to this configurercollect {@link ValidationFailure}s. */ @Beta public *interface @paramFailureCollector message{ failure message /** @param correctiveAction corrective action * @returnAdd a validation failure to */this ValidationFailure addFailure(String message, @Nullable String correctiveAction); /** * Throws validation exception if there are any failures that are added to the configurer through * {@link #addFailure(String, String)}. * * @throws ValidationException if there are any validation failures */ void throwIfFailure() throws ValidationException; |
Code Block | ||||
---|---|---|---|---|
| ||||
/**
* Validation exception that carries multiple validation failures.
*/
@Beta
public class ValidationException extends RuntimeException {
private List<ValidationFailure> failures;
/**
* Creates a validation exception with list of failures.
*
* @param failures list of validation failures
*/
public ValidationException(List<ValidationFailure> failures) {
super(failures.isEmpty() ? "Validation Exception occurred." : failures.iterator().next().getMessage());
this.failures = failures;
}
/**
* Returns a list of validation failures.
*/
public List<ValidationFailure> getFailures() {
return failures;
}
} |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Represents a failure condition occurred during validation. */ @Beta public class ValidationFailure { private final String message; private final String correctiveAction; private final List<Cause> causes; /** * Creates a validation failure with provided message and corrective action. * * @param message validation failure message * @param correctiveAction corrective action */ public ValidationFailure(String message, String correctiveAction) { this.message = message; this.correctiveAction = correctiveAction; this.causes = new ArrayList<>(); } /** * Adds provided cause to this validation failure. * * @param cause cause of validation failure * @return validation failure with provided cause */ public ValidationFailure withCause(Cause cause) { causes.add(cause); return this; }failure collector. The method returns the validation failure that was added to * the failure collector. This failure can be used to add additional {@link ValidationFailure.Cause}s. * For example, * <code>failureCollector.addFailure("message", "action").withConfigProperty("configProperty");</code> * * @param message failure message * @param correctiveAction corrective action * @return a validation failure * @throws UnsupportedOperationException if the implementation does not override this method */ default ValidationFailure addFailure(String message, @Nullable String correctiveAction) { throw new UnsupportedOperationException("Adding a failure is not supported."); } /** * Throws validation exception if there are any failures that are added to the failure collector through * {@link #addFailure(String, String)}. * If no failures are added to the collector, it will return a {@link ValidationException} with empty failure list. * * <pre> * String someMethod() { * switch (someVar) { * // cases * } * // if control comes here, it means failure * failureCollector.addFailure(...); * // throw validation exception so that compiler knows that exception is being thrown which eliminates the need to * // have a statement that returns null towards the end of this method * throw failureCollector.getOrThrowException(); * } * </pre> * * @return returns a {@link ValidationException} if no failures were added to the collector * @throws ValidationException exception indicating validation failures * @throws UnsupportedOperationException if the implementation does not override this method */ default ValidationException getOrThrowException() throws ValidationException { throw new UnsupportedOperationException("Throwing failures is not supported."); } } |
Code Block |
---|
public interface StageConfigurer { .... /** * AddsReturns causea attributesfailure thatcollector representsfor plugin not found failure causethe stage. * * @param@return pluginIda pluginfailure idcollector * @param@throws pluginNameUnsupportedOperationException pluginif namethe implementation does not *override @paramthis pluginTypemethod plugin type */ * @returndefault validationFailureCollector failure with plugin not found causegetFailureCollector() { throw */ public ValidationFailure withPluginNotFoundCause(String pluginId, String pluginName, String pluginType) { causes.add(new Cause().with(CauseAttributes.PLUGIN_ID, pluginId).with(CauseAttributes.PLUGIN_NAME, pluginName) .with(CauseAttributes.PLUGIN_TYPE, pluginType)); return this; } new UnsupportedOperationException("Getting failure collector is not supported."); } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Validation exception that carries multiple validation failures. */ @Beta public class ValidationException extends RuntimeException { private final List<ValidationFailure> failures; /** * AddsCreates causea attributesvalidation thatexception representswith pluginlist configureof failure causefailures. * * @param pluginConfigfailures list pluginof configvalidation propertyfailures */ @return validation failurepublic with invalid plugin config property cause */ public ValidationFailure withPluginConfigCause(String pluginConfig) { causes.add(new Cause().with(CauseAttributes.PLUGIN_CONFIG, pluginConfig)); return thisValidationException(List<ValidationFailure> failures) { super("Errors were encountered during validation."); this.failures = Collections.unmodifiableList(new ArrayList<>(failures)); } /** * AddsReturns causea attributeslist thatof represents invalid input schema field failure causevalidation failures. */ public *List<ValidationFailure> @paramgetFailures() fieldName{ name of the input schemareturn fieldfailures; } } |
Code Block | ||||
---|---|---|---|---|
| ||||
/** @param inputStage* stageRepresents namea failure condition occurred *during @return validation. failure with*/ invalid@Beta inputpublic schemaclass fieldValidationFailure cause{ private static */final Gson GSON public= ValidationFailurenew withInvalidInputSchemaCause(String fieldName, @NullableGson(); private final String inputStage)message; { private final causes.add(new Cause().with(CauseAttributes.INPUT_STAGE, inputStage) String correctiveAction; private final List<Cause> causes; /** * Creates a validation failure with provided .with(CauseAttributes.INPUT_SCHEMA_FIELD, fieldName));message. * return this; * }@param message validation failure /**message */ Adds cause attributespublic that represents invalid output schema field failure cause. *ValidationFailure(String message) { this(message, null); } /** @param fieldName* nameCreates ofa thevalidation outputfailure schemawith fieldprovided message and corrective *action. @param outputPort stage name* * @return@param message validation failure with invalid output schema field causemessage * @param correctiveAction corrective action */ public ValidationFailure withInvalidOutputSchemaCause(String fieldNamemessage, @Nullable String outputPortcorrectiveAction) { causesthis.add(new Cause().with(CauseAttributes.OUTPUT_PORT, outputPort)message = message; this.correctiveAction = correctiveAction; this.causes = new .with(CauseAttributes.OUTPUT_SCHEMA_FIELD, fieldName)); return thisArrayList<>(); } /** * ReturnsAdds failureprovided message.cause to this validation */failure. public String* getMessage() { * @param cause returncause message;of validation }failure /** @return validation failure *with Returnsprovided correctivecause action for this failure.*/ public */ValidationFailure withCause(Cause cause) @Nullable{ public String getCorrectiveActioncauses.add(cause); { return correctiveActionthis; } /** * ReturnsAdds cause causesattributes that causedrepresents thisplugin not found failure cause. */ * public@param List<Cause>pluginId getCauses()plugin {id * return@param causes;pluginName plugin name } * @param @OverridepluginType plugin type public boolean equals(Object o)* {@return validation failure with plugin ifnot (thisfound ==cause o) { */ public ValidationFailure return true; }withPluginNotFound(String pluginId, String pluginName, String pluginType) { ifreturn withPluginNotFound(o ==pluginId, pluginName, pluginType, null, || getClass() != o.getClass()) {null); } /** return* false;Adds cause attributes that represents }plugin not found failure cause. ValidationFailure failure = (ValidationFailure)* o; * @param return message.equals(failure.message) &&pluginId plugin id * @param pluginName Objects.equals(correctiveAction, failure.correctiveAction) && causes.equals(failure.causes); } @Override public int hashCode() {plugin name * @param pluginType plugin type * @param requestedArtifact requested artifact * return Objects.hash(message, correctiveAction, causes);@param suggestedArtifact suggested artifact } * @return validation /**failure with plugin not * Represents afound cause of a failure. */ @Betapublic ValidationFailure withPluginNotFound(String publicpluginId, staticString classpluginName, CauseString {pluginType, private final Map<String, String> attributes; /** * Creates a failure cause. */ public Cause() { this.attributes = new HashMap<>(); @Nullable ArtifactId requestedArtifact, } /** * Adds attributes to this cause. * * @param attribute cause attribute name * @param value cause@Nullable attributeArtifactId valuesuggestedArtifact) { * @returnCause this cause */ public Cause with(String attribute, String value) {= new Cause().addAttribute(CauseAttributes.PLUGIN_ID, pluginId) attributes.put(attributeaddAttribute(CauseAttributes.PLUGIN_NAME, valuepluginName); return this.addAttribute(CauseAttributes.PLUGIN_TYPE, pluginType); }if (requestedArtifact != null) { /** * Returns cause attributes.cause.addAttribute(CauseAttributes.REQUESTED_ARTIFACT_NAME, requestedArtifact.getName()); */ public Map<String, String> getAttributes() { cause.addAttribute(CauseAttributes.REQUESTED_ARTIFACT_SCOPE, requestedArtifact.getScope().name()); return attributescause.addAttribute(CauseAttributes.REQUESTED_ARTIFACT_VERSION, requestedArtifact.getVersion().getVersion()); } @Override public boolean equals(Object o) { if (thissuggestedArtifact !== onull) { return truecause.addAttribute(CauseAttributes.SUGGESTED_ARTIFACT_NAME, suggestedArtifact.getName()); } cause.addAttribute(CauseAttributes.SUGGESTED_ARTIFACT_SCOPE, suggestedArtifact.getScope().name()); if (o == null || getClass() != o.getClass cause.addAttribute(CauseAttributes.SUGGESTED_ARTIFACT_VERSION, suggestedArtifact.getVersion().getVersion()); { } return false;causes.add(cause); return this; } /** Cause cause* =Adds (Cause) o; return attributes.equals(cause.attributes);cause attributes that represents invalid stage configure property failure cause. * } * @param stageConfigProperty @Overridestage config property public int* hashCode()@return {validation failure with invalid stage config property return Objects.hash(attributes);cause */ } public } } |
All the attributes of a cause can be tracked at central location as below:
Code Block | ||||
---|---|---|---|---|
| ||||
/**
* Cause attributes constants.
*/
@Beta
public final class CauseAttributes {
// represents plugin configuration property failure
public static final String PLUGIN_CONFIG = "pluginConfig";
// represents id of the plugin
public static final String PLUGIN_ID = "pluginId";
// represents type of the plugin
public static final String PLUGIN_TYPE = "pluginType";
// represents name of the plugin
public static final String PLUGIN_NAME = "pluginName";
// represents input stage
public static final String INPUT_STAGE = "inputStage";
// represents field of input stage schema
public static final String INPUT_SCHEMA_FIELD = "inputField";
// represents a port of stage output
public static final String OUTPUT_PORT = "outputPort";
// represents a field of output port schema
public static final String OUTPUT_SCHEMA_FIELD = "outputField";
private CauseAttributes() {
// no-op
}
} |
API usage in plugins
Code Block |
---|
@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class); StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer(); try { Pattern.compile(conf.filterRegex); } catch (Exception e) { // add validation error to stage configurer stageConfigurer.addFailure(new ValidationFailure(e.getMessage(), "Provide a valid regex for property 'filterRegex'.").withPluginConfigCause("filterRegex")); } if (conf.sourceFileset.equals(conf.destinationFileset)) { // add validation error to stage configurer stageConfigurer.addFailure(new ValidationFailure("source and destination filesets must be different", "Provide different fileset for source and destination.") .withPluginConfigCause("sourceFileset").withPluginConfigCause("sourceFileset"); stageConfigurer.addValidationFailure(new InvalidStageFailure("sourceValidationFailure withConfigProperty(String stageConfigProperty) { causes.add(new Cause().addAttribute(CauseAttributes.STAGE_CONFIG, stageConfigProperty)); return this; } /** * Adds cause attributes for failure cause that represents an invalid element in the list associated with given stage * configure property. * * @param stageConfigProperty stage config property * @param element element in the list associated by a given stageConfigProperty * @return validation failure with invalid stage config property element cause */ public ValidationFailure withConfigElement(String stageConfigProperty, String element) { causes.add(new Cause().addAttribute(CauseAttributes.STAGE_CONFIG, stageConfigProperty) .addAttribute(CauseAttributes.CONFIG_ELEMENT, element)); return this; } /** * Adds cause attributes that represents invalid input schema field failure cause. * * @param fieldName name of the input schema field * @param inputStage stage name * @return validation failure with invalid input schema field cause */ public ValidationFailure withInputSchemaField(String fieldName, @Nullable String inputStage) { Cause cause = new Cause().addAttribute(CauseAttributes.INPUT_SCHEMA_FIELD, fieldName); cause = inputStage == null ? cause : cause.addAttribute(CauseAttributes.INPUT_STAGE, inputStage); causes.add(cause); return this; } /** * Adds cause attributes that represents invalid output schema field failure cause. * * @param fieldName name of the output schema field * @param outputPort stage name * @return validation failure with invalid output schema field cause */ public ValidationFailure withOutputSchemaField(String fieldName, @Nullable String outputPort) { Cause cause = new Cause().addAttribute(CauseAttributes.OUTPUT_SCHEMA_FIELD, fieldName); cause = outputPort == null ? cause : cause.addAttribute(CauseAttributes.OUTPUT_PORT, outputPort); causes.add(cause); return this; } /** * Adds cause attributes that represents a stacktrace. * * @param stacktraceElements stacktrace for the error * @return validation failure with stacktrace */ public ValidationFailure withStacktrace(StackTraceElement[] stacktraceElements) { causes.add(new Cause().addAttribute(CauseAttributes.STACKTRACE, GSON.toJson(stacktraceElements))); return this; } /** * Returns failure message. */ public String getMessage() { return message; } /** * Returns corrective action for this failure. */ @Nullable public String getCorrectiveAction() { return correctiveAction; } /** * Returns causes that caused this failure. */ public List<Cause> getCauses() { return causes; } @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } ValidationFailure failure = (ValidationFailure) o; return message.equals(failure.message) && Objects.equals(correctiveAction, failure.correctiveAction) && causes.equals(failure.causes); } @Override public int hashCode() { return Objects.hash(message, correctiveAction, causes); } /** * Represents a cause of a failure. */ @Beta public static class Cause { private final Map<String, String> attributes; /** * Creates a failure cause. */ public Cause() { this.attributes = new HashMap<>(); } /** * Adds an attribute to this cause. * * @param attribute cause attribute name * @param value cause attribute value * @return this cause */ public Cause addAttribute(String attribute, String value) { attributes.put(attribute, value); return this; } /** * Returns value of the provided cause attribute. * * @param attribute attribute name */ public String getAttribute(String attribute) { return attributes.get(attribute); } /** * Returns all the attributes of the cause. */ public Map<String, String> getAttributes() { return Collections.unmodifiableMap(new HashMap<>(attributes)); } @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Cause cause = (Cause) o; return attributes.equals(cause.attributes); } @Override public int hashCode() { return Objects.hash(attributes); } } } |
All the attributes of a cause can be tracked at central location as below:
Code Block | ||||
---|---|---|---|---|
| ||||
/**
* Cause attributes constants.
*/
@Beta
public class CauseAttributes {
// Represents stage configuration property failure
public static final String STAGE_CONFIG = "stageConfig";
// Represents an element in the list of elements associated with a stage config property.
// For example, in projection transform, config property 'keep' represents a list of input fields to keep. Below
// cause attribute can be used to represent an invalid field in 'keep' config property
public static final String CONFIG_ELEMENT = "configElement";
// Represents id of the plugin
public static final String PLUGIN_ID = "pluginId";
// Represents type of the plugin
public static final String PLUGIN_TYPE = "pluginType";
// Represents name of the plugin
public static final String PLUGIN_NAME = "pluginName";
// Represents requested artifact name
public static final String REQUESTED_ARTIFACT_NAME = "requestedArtifactName";
// Represents requested artifact name
public static final String REQUESTED_ARTIFACT_VERSION = "requestedArtifactVersion";
// Represents requested artifact scope
public static final String REQUESTED_ARTIFACT_SCOPE = "requestedArtifactScope";
// Represents suggested artifact name
public static final String SUGGESTED_ARTIFACT_NAME = "suggestedArtifactName";
// Represents suggested artifact name
public static final String SUGGESTED_ARTIFACT_VERSION = "suggestedArtifactVersion";
// Represents suggested artifact scope
public static final String SUGGESTED_ARTIFACT_SCOPE = "suggestedArtifactScope";
// Represents input stage
public static final String INPUT_STAGE = "inputStage";
// represents field of input stage schema
public static final String INPUT_SCHEMA_FIELD = "inputField";
// Represents a stage output port
public static final String OUTPUT_PORT = "outputPort";
// Represents a field of output stage schema
public static final String OUTPUT_SCHEMA_FIELD = "outputField";
// Represents a stacktrace
public static final String STACKTRACE = "stacktrace";
} |
API usage in plugins
Code Block | ||
---|---|---|
| ||
@Override public void configurePipeline(PipelineConfigurer pipelineConfigurer) { pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class); FailureCollector collector = pipelineConfigurer.getStageConfigurer().getFailureCollector(); try { Pattern.compile(conf.filterRegex); } catch (Exception e) { collector.addFailure("Error encountered while compiling filter regex: " + e.getMessage(), "Make sure filter regex is valid.").withConfigProperty("filterRegex"); } if (conf.sourceFileset.equals(conf.destinationFileset)) { collector.addFailure("Source and destination filesets must be different", stageName)); } } |
Impact on UI
"failures": [
{
|
"message":
|
|
|
|
|
|
|
|
|
|
|
|
" |
Make sure source and destination filesets |
are different") |
.withConfigProperty("sourceFileset").withConfigProperty("destinationFileset"); } |
}
]
}
} |
Impact on UI
Type | Description | Scenario | Approach - 1 - Json Response | Approach - 2 - Json Response | ||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
StageError | Represents validation error while configuring the stage | If there is any error while connecting to sink while getting actual schema | { | errorsfailures": [ | : "Make sure correct driver is | uploadedavailable.", | causesproperties": | [ | { | "stage": "src" | } | ]} | InvalidProperty | Represents invalid configuration property | If config property value contains characters that are not allowed by underlying source or sink | { | failureserrors": [ | "type": "InvalidProperty" | ,message": " | CanCould not | specifyload | bothjdbc | dropdriver | and keepclass.", "correctiveAction" : " | EitherMake | dropsure | orcorrect | keepdriver | should be emptyis available.", | propertiescauses": [ | projectionsrc" | ,
| "configProperty": "drop"] | |
InvalidProperty | Represents invalid configuration property | If config property value contains characters that are not allowed by underlying source or sink | { | { "correctiveAction" : "EitherMake dropsure or'millis' keepis should be empty", | ||||||||||||||||||||||||||||||
PluginNotFound | Represents plugin not found error for a stage. This error will be added by the data pipeline app | If the plugin was not found. This error will be thrown from the data pipeline app | { | { "correctiveAction" : "Please make sure the 'Mock' plugin is installed.", "pluginId" : "Mock" | ||||||||||||||||||||||||||||||
InvalidInputSchema | Represents invalid schema field in input schema | If the input schemas for joiner plugin is of different types | { ] | { "properties": { "field"inputStage": "namesource1", } | ||||||||||||||||||||||||||||||
InvalidOutputSchema | Represents invalid schema field in output schema | If the output schema for the plugin is not compatible with underlying sink | { | { "correctiveAction" : "ChangeSchema theshould typebe of the schema fieldtype 'namestring' at tooutput beport 'stringport'", | ||||||||||||||||||||||||||||||
InvalidOutputSchemaInvalidFieldInProperty | Represents an invalid schema field in output schemaproperty list | If the output schema for the plugin is not compatible with underlying sinkthe property represents list of fields, the failure should include the property name along with invalid field | { | { |
Conclusion
There are 2 contracts in this design. Programmatic contract between data pipeline app and plugins and another between data pipeline app and UI. Approach 2 does not introduce concept of failure type. This means that contract with UI will be based on the cause attributes rather than the type. This means that if plugins creates a custom failure and uses any of the UI compatible attributes, the UI can still highlight them. Approach 2 also provides association between causes which represents the failure better in case there are multiple causes causing this failure. Hence, Approach 2 is suggested.
Related Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Related Work
- Improvements in to Error HandlingMessages
- Pipeline System Application