Introduction
CDAP pipeline is composed of various plugins that can be configured by users as CDAP pipelines are being developed. While building CDAP pipelines, pipeline developer can provide invalid plugin configurations. For example, the BigQuery sink plugin can have schema which does not match with an underlying BigQuery table. CDAP pipeline developer can use new validation endpoint to validate the stages before deploying the pipeline. In order to fail fast, validation endpoint should return all the validation errors from a given stage when this endpoint is called.
Data pipeline app exposes various error types for plugin validation. In future releases, new error types can be introduced. When plugins with new error types are pushed to hub, data pipeline artifacts need to be upgraded for every new type of error that is introduced. This is because the validation errors are defined in the data pipeline app itself. A better approach would be to modify data pipeline app so that app artifacts do not need to be replaced for every new type of error.
Goals
To fail fast, introduce a new api to collect multiple error messages from plugins at configure time
- Decouple various validation error types from data pipeline app
Instrument plugins to use this api to return multiple error messages for validation endpoint
User Stories
- As a CDAP pipeline developer, when I validate a stage, I expect that all the invalid config properties and schema fields are highlighted on CDAP UI with appropriate error message.
- As a plugin developer, I should be able to specify all the validation errors while configuring the plugin for better user experience.
- As a plugin developer, I should be able to use new validation error types without replacing data pipeline app artifacts.
Scenarios
- Sources
- Transforms
- Sinks
- Joiner
- Splitter
API Changes for Plugin Validation
Collect Multiple errors from plugins
To collect multiple stage validation errors from the stage, StageConfigurer, MultiInputStageConfigurer and MultiOutputStageConfigurer can be modified as below. If there are any validation errors added to stage configurer, the pipeline deployment will fail and all the errors will be returned as a response to stage validation REST endpoint.
Current implementation does not expose stage name to the plugin in configurePipeline method. Stage name will be needed by the plugins to create stage specific errors. For that stageName will be exposed to through stage configurer as below.
Code Block |
---|
language | java |
---|
title | StageConfigurer.java |
---|
|
public interface StageConfigurer {
...
/**
* get the stage name.
* @return stage name
*/
String getStageName();
/**
* add validation errors.
* @param e errors
*/
void addValidationError(ValidationError e); |
Decouple plugin error types from data pipeline app
Approach - 1
A new ValidationError class is introduced to collect multiple validation errors in stage configurer. This class will be exposed by data pipeline app to the plugins. Each new type of error can be added to hydrator-common which can be added as compile time dependency to the plugins. This approach allows us to easily add more types of validation errors for plugins while removing a need to update data pipeline artifact for each new error type. Below are some of the error types that will be added to hydrator-common module.
Code Block |
---|
language | java |
---|
title | ValidationError.java |
---|
|
/**
* Represents error that occurred during validation.
*/
@Beta
public class ValidationError {
private final String message;
private final String type;
/**
* Creates an error with provided error message.
* @param message error message
*/
public ValidationError(String message) {
this.message = message;
this.type = getType();
}
/**
* Returns the type of the error.
*/
public String getType() {
return "ERROR";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
ValidationError error = (ValidationError) o;
return message.equals(error.message) &&
type.equals(error.type);
}
@Override
public int hashCode() {
return Objects.hash(message, type);
}
} |
Introduced Errors
Following errors will be added to hydrator-common. Note that this list will keep evolving as new types of errors are added to the module.
Code Block |
---|
language | java |
---|
title | InvalidStageError.java |
---|
|
/**
* Represents error that occurred during stage validation.
*/
@Beta
public class InvalidStageError extends ValidationError {
private final String stage;
/**
* Creates error that represents a stage validation error.
*
* @param message error message
* @param stage name of the stage
*/
public InvalidStageError(String message, String stage) {
super(message);
this.stage = stage;
}
@Override
public String getType() {
return "STAGE_ERROR";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
if (!super.equals(o)) {
return false;
}
InvalidStageError that = (InvalidStageError) o;
return Objects.equals(stage, that.stage);
}
@Override
public int hashCode() {
return Objects.hash(super.hashCode(), stage);
}
} |
Code Block |
---|
language | java |
---|
title | InvalidStagePropertyError.java |
---|
|
/**
* Represents invalid stage property error that occurred during stage validation.
*/
@Beta
public class InvalidStagePropertyError extends InvalidStageError {
private final String property;
/**
* Creates error that represents invalid stage property.
*
* @param message error message
* @param stage name of the stage
* @param property invalid property
*/
public InvalidStagePropertyError(String message, String stage, String property) {
super(message, stage);
this.property = property;
}
@Override
public String getType() {
return "INVALID_FIELD";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
if (!super.equals(o)) {
return truefalse;
}
InvalidStagePropertyError that if= (InvalidStagePropertyError) o;
== null || getClass() !=return oproperty.getClassequals()that.property);
{ }
@Override
return false; public int hashCode() {
} ValidationError error = (ValidationError) oreturn Objects.hash(super.hashCode(), property);
}
return message.equals(error.message) &&
type.equals(error.type);
}
@Override
public int hashCode() {
return Objects.hash(message, type);
}
} |
Introduced Errors
Following errors will be added to hydrator-common. Note that this list will keep evolving as new types of errors are added to the module.
Code Block |
---|
language | java |
---|
title | InvalidStageError.java |
---|
|
/**
* Represents error that occurred during stage validation.
*/
@Beta
public class InvalidStageError extends ValidationError {
private final String stage;
/**
* Creates error that represents a stage validation error.
*
* @param message error message
* @param stage name of the stage
*/
public InvalidStageError(String message, String stage) {
super(message);
this.stage = stage;
}
@Override
public String getType() {
return "STAGE_ERROR";
}
@Override
public boolean equals(Object o) {
if (this == o)}
|
Code Block |
---|
language | java |
---|
title | InvalidInputSchemaFieldError.java |
---|
|
/**
* Represents invalid schema field in input schema.
*/
public class InvalidInputSchemaFieldError extends InvalidStageError {
private final String field;
/**
* Creates error that represents invalid schema field in input schema.
* @param message error message
* @param stage name of the stage
* @param field field that is invalid in input schema
*/
public InvalidInputSchemaFieldError(String message, String stage, String field) {
super(message, stage);
this.field = field;
}
@Override
public String getType() {
return "INVALID_INPUT_SCHEMA";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
if (!super.equals(o)) {
return truefalse;
}
ifInvalidInputSchemaFieldError that = (InvalidInputSchemaFieldError) o;
== null || getClass() !=return ofield.getClassequals(that.field));
{ }
@Override
return false; public int hashCode() {
} return if Objects.hash(!super.equalshashCode(o), field);
{ }
} |
Code Block |
---|
language | java |
---|
title | InvalidOutputSchemaFieldError.java |
---|
|
/**
* Represents invalid returnschema false;field in output schema.
}*/
public class InvalidOutputSchemaFieldError extends InvalidStageError that{
= (InvalidStageError) o;private final String field;
return Objects.equals(stage, that.stage); /**
} * Creates error @Overridethat represents invalid publicschema intfield hashCode()in {output schema.
return Objects.hash(super.hashCode(), stage);
}
} |
Code Block |
---|
language | java |
---|
title | InvalidStagePropertyError.java |
---|
|
/**
* Represents invalid stage property error that occurred during stage validation.
*/
@Beta
public class InvalidStagePropertyError extends InvalidStageError {
private final String property;
/**
* Creates error that represents invalid stage property.
*
* @param message error message
* @param stage name of the stage
* @param property invalid property
*/
public InvalidStagePropertyError(String message, String stage, String property* @param message error message
* @param stage name of the stage
* @param field field that is invalid in output schema
*/
public InvalidOutputSchemaFieldError(String message, String stage, String field) {
super(message, stage);
this.field = field;
}
@Override
public String getType() {
return "INVALID_OUTPUT_SCHEMA";
}
@Override
public boolean equals(Object o) {
if (this == o) {
super(message, stage); return true;
this.property = property;}
} if (o @Override== null || public String getTypegetClass() != o.getClass()) {
return "INVALID_FIELD"false;
} }
@Override public boolean if (!super.equals(Object o)) {
return if (this == o) {false;
}
InvalidOutputSchemaFieldError that = return(InvalidOutputSchemaFieldError) trueo;
}return field.equals(that.field);
}
if (o ==@Override
null || getClass() != o.getClasspublic int hashCode()) {
return return false;
}
if (!super.equals(o)) {
return false;
}
InvalidStagePropertyError that = (InvalidStagePropertyError) o;
return property.equals(that.property);
}
@Override
public int hashCode() {
return Objects.hash(super.hashCode(), propertyObjects.hash(super.hashCode(), field);
}
} |
API usage in plugins
Code Block |
---|
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class);
StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
// get the name of the stage
String stageName = stageConfigurer.getStageName();
try {
Pattern.compile(conf.filterRegex);
} catch (Exception e) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStagePropertyError(e.getMessage(), stageName, "filterRegex"));
}
} |
Code Block |
---|
language | java |
---|
title | InvalidInputSchemaFieldError.java |
---|
|
/**
* Represents invalid schema field in input schema.
*/
public class InvalidInputSchemaFieldError extends InvalidStageError {
private final String field;
/**
* Creates error that represents invalid schema field in input schema.
* @param message error message
* @param stage name of the stage
* @param field field that is invalid in input schema
*/
public InvalidInputSchemaFieldError(String message, String stage, String field) {
super(message, stage);
this.field = field;
}
@Override
public String getType() {
return "INVALID_INPUT_SCHEMA";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
if (!super.equals(o)) {
return false;
}
InvalidInputSchemaFieldError that = (InvalidInputSchemaFieldError) o;
return field.equals(that.field);
}
@Override
public int hashCode() {
return Objects.hash(super.hashCode(), field);
}
} |
Code Block |
---|
language | java |
---|
title | InvalidOutputSchemaFieldError.java |
---|
|
/**
* Represents invalid schema field in output schema.
*/
public class InvalidOutputSchemaFieldError extends InvalidStageError {
private final String field;
/**
* Creates error that represents invalid schema field in output schema.
* @param message error message
* @param stage name of the stage
* @param field field that is invalid in output schema
*/
public InvalidOutputSchemaFieldError(String message, String stage, String field) {
super(message, stage);
this.field = field;
}
@Override
public String getType() {
return "INVALID_OUTPUT_SCHEMA";
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
if (!super.equals(o)) {
return false;
}
InvalidOutputSchemaFieldError that = (InvalidOutputSchemaFieldError) o;
return field.equals(that.field);
}
@Override
public int hashCode() {
return Objects.hash(super.hashCode(), field);
}
} |
API usage in plugins
Code Block |
---|
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class);
StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
// get the name of the stage
String stageName = stageConfigurer.getStageName();
try {
Pattern.compile(conf.filterRegex);
} catch (Exception e) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStagePropertyError(e.getMessage(), stageName, "filterRegex"));
}
if (conf.sourceFileset.equals(conf.destinationFileset)) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStageError("source and destination filesets must be different", stageName));
}
} |
Impact on UI
UI should be able to handle new error types that are introduced. For example, for invalid stage properties, UI should highlight all the invalid properties for a given stage. For schema mismatch, UI should be able to highlight schema fields that are mismatching.
Below are the responses to the validation endpoint for each type of validation error:
Type | Description | Scenario | Json Response |
---|
STAGE_ERROR | Represents validation error while configuring the stage | If there is any error while connecting to sink while getting actual schema | { "errors": [ { "type" : "STAGE_ERROR",
"stage" : "src",
"message" : "source and destination filesets must be different" } ] } |
INVALID_PROPERTY | Represents invalid configuration property | If config property value contains characters that are not allowed by underlying source or sink | { "errors": [ { "type" : "INVALID_PROPERTY",
"stage" : "src",
"message" : "Invalid config for property 'port'", "property" : "port" } ] } |
PLUGIN_NOT_FOUND | Represents plugin not found error for a stage | If the plugin was not found. This error will be thrown from the data pipeline app | { "errors": [ { "stage": "src", "type": "PLUGIN_NOT_FOUND", "message": "Plugin named 'Mock' of type 'batchsource' not found.", "pluginType": "batchsource", "pluginName": "Mock", "requestedArtifact": { "scope": "USER", "name": "app-mocks-ghost", "version": "1.0.0" }, "suggestedArtifact": { "scope": "USER", "name": "app-mocks", "version": "1.0.0" } } ] } |
INVALID_INPUT_SCHEMA | Represents invalid schema field in input schema | If the input schema for the plugin is not supported by underlying plugin | { "errors": [ { "type" : "INVALID_INPUT_SCHEMA",
"stage" : "sink",
"message" : "Invalid schema field 'email'.", "field" : "email" } ] } |
INVALID_OUTPUT_SCHEMA | Represents invalid schema field in output schema | If the output schema for the plugin is not compatible with underlying sink | { "errors": [ { "type" : "INVALID_OUTPUT_SCHEMA",
"stage" : "sink",
"message" : "Invalid schema field 'email'.", "field" : "email" } ] } |
Approach - 2
Validation error represents an error with various causes with extra properties that caused the error. For example, when the input schema field type does not match the underlying sink schema, information such as error message, stage name, field name, suggested type will be useful information to have in the validation error. Each error message can be associated to more than one causes. This can happen for plugins such as joiner and splitter where there are multiple input or output schemas from a given stage. For example, when input schemas for joiner are not compatible, the validation error information should contain error message along with the fields from both the stages that caused this error. This means that a validation error can be represented as a list of causes where each cause is a map of string to object as shown below.
Code Block |
---|
language | java |
---|
title | ValidationError.java |
---|
|
/**
* Represents error that occurred during validation.
*/
@Beta
public class ValidationError {
private final String message;
protected final List<Map<String, Object>> causes;
/**
* Creates a validation error with a message and empty map of causes
* @param message
*/
public ValidationError(String message) {
this.message = message;
this.causes = new ArrayList<>();
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
ValidationError that = (ValidationError) o;
return message.equals(that.message) && causes.equals(that.causes);
}
@Override
public int hashCode() {
return Objects.hash(message, causes);
}
} |
All the attributes of an error can be tracked at central location as below:
Code Block |
---|
language | java |
---|
title | ErrorAttributes.java |
---|
|
/**
* Error attributes.
*/
public enum ErrorAttributes {
STAGE("stage"), // represents stage being validated
FIELD("field"), // represents input/output schema field
PROPERTY("property"), // represents stage property
OUTPUT_PORT("output_port"), // represents output port for plugins such as SplitterTransform where multiple output schemas are expected
INPUT_STAGE("input_stage"), // represents input stage for plugins such as Joiner where multiple input schemas are expected
..
private String name;
ErrorAttributes(String name) {
this.name = name;
}
} |
Introduced Errors
With this approach following error classes can be added to hydrator-common which represents specific type of errors.
Code Block |
---|
language | java |
---|
title | InvalidStageError.java |
---|
|
/**
* Represents error that occurred during stage validation.
*/
@Beta
public class InvalidStageError extends ValidationError {
/**
* Creates validation error that occurred during stage validation.
* @param message error message
* @param stage name of the stage that caused this validation error
*/
public InvalidStageError(String message, String stage) {
super(message);
causes.add(Collections.singletonMap("stage", stage));
}
} |
Code Block |
---|
language | java |
---|
title | InvalidStagePropertyError.java |
---|
|
/**
* Represents error that occurred during stage config property validation.
*/
@Beta
public class InvalidStagePropertyError extends ValidationError {
/**
* Creates validation error that occurred during stage validation.
* @param message error message
* @param stage name of the stage that caused this validation error
* @param property property that is invalid
*/
public InvalidStageError(String message, String stage, String property) {
super(message);
Map<String, Object> map = new HashMap<>();
map.put("stage", stage);
map.put("property", property);
causes.add(map);
}
} |
Code Block |
---|
/**
* Represents invalid input schema error.
*/
public class InvalidInputSchemaError extends NewValidationError {
/**
* Creates invalid input schema error.
* @param message error message
* @param stage name of the stage
* @param map map of incoming stage name to field that is invalid.
*/
public InvalidInputSchemaError(String message, String stage, Map<String, String> map) {
super(message);
for (Map.Entry<String, String> entry : map.entrySet()) {
Map<String, Object> causeMap = new HashMap<>();
causeMap.put("stage", stage);
causeMap.put("input_stage", entry.getKey());
causeMap.put("field", entry.getValue());
causes.add(causeMap);
}
}
} |
Code Block |
---|
language | java |
---|
title | InvalidOutputSchemaError.java |
---|
|
/**
* Represents invalid output schema error.
*/
public class InvalidOutputSchemaError extends NewValidationError {
/**
* Creates invalid output schema error.
* @param message error message
* @param stage name of the stage
* @param map map of output going port name to field that is invalid
*/
public InvalidOutputSchemaError(String message, String stage, Map<String, String> map) {
super(message);
for (Map.Entry<String, String> entry : map.entrySet()) {
Map<String, Object> causeMap = new HashMap<>();
causeMap.put("stage", stage);
causeMap.put("output_port", entry.getKey());
causeMap.put("field", entry.getValue());
causes.add(causeMap);
}
}
} |
API usage in plugins
The above api usage in the plugins is same as approach-1.
Code Block |
---|
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class);
StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
// get the name of the stage
String stageName = stageConfigurer.getStageName();
try {
Pattern.compile(conf.filterRegex);
} catch (Exception e) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStagePropertyError(e.getMessage(), stageName, "filterRegex"));
}
if (conf.sourceFileset.equals(conf.destinationFileset)) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStageError("source and destination filesets must be different", stageName));
}
} |
Impact on UI
List of exceptions
Conclusion
Approach if (conf.sourceFileset.equals(conf.destinationFileset)) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStageError("source and destination filesets must be different", stageName));
}
} |
Approach - 2
Validation error represents an error with various causes with extra properties that caused the error. For example, when the input schema field type does not match the underlying sink schema, information such as error message, stage name, field name, suggested type will be useful information to have in the validation error. Each error message can be associated to more than one causes. This can happen for plugins such as joiner and splitter where there are multiple input or output schemas from a given stage. For example, when input schemas for joiner are not compatible, the validation error information should contain error message along with the fields from both the stages that caused this error. This means that a validation error can be represented as a list of causes where each cause is a map of string to object as shown below.
Code Block |
---|
language | java |
---|
title | ValidationError.java |
---|
|
/**
* Represents error that occurred during validation.
*/
@Beta
public class ValidationError {
private final String message;
protected final List<Map<String, Object>> causes;
/**
* Creates a validation error with a message and empty map of causes
* @param message
*/
public ValidationError(String message) {
this.message = message;
this.causes = new ArrayList<>();
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
ValidationError that = (ValidationError) o;
return message.equals(that.message) && causes.equals(that.causes);
}
@Override
public int hashCode() {
return Objects.hash(message, causes);
}
} |
All the attributes of an error can be tracked at central location as below:
Code Block |
---|
language | java |
---|
title | ErrorAttributes.java |
---|
|
/**
* Error attributes.
*/
public enum ErrorAttributes {
STAGE("stage"), // represents stage being validated
FIELD("field"), // represents input/output schema field
PROPERTY("property"), // represents stage property
OUTPUT_PORT("output_port"), // represents output port for plugins such as SplitterTransform where multiple output schemas are expected
INPUT_STAGE("input_stage"), // represents input stage for plugins such as Joiner where multiple input schemas are expected
..
private String name;
ErrorAttributes(String name) {
this.name = name;
}
} |
Introduced Errors
With this approach following error classes can be added to hydrator-common which represents specific type of errors.
Code Block |
---|
language | java |
---|
title | InvalidStageError.java |
---|
|
/**
* Represents error that occurred during stage validation.
*/
@Beta
public class InvalidStageError extends ValidationError {
/**
* Creates validation error that occurred during stage validation.
* @param message error message
* @param stage name of the stage that caused this validation error
*/
public InvalidStageError(String message, String stage) {
super(message);
causes.add(Collections.singletonMap("stage", stage));
}
} |
Code Block |
---|
language | java |
---|
title | InvalidStagePropertyError.java |
---|
|
/**
* Represents error that occurred during stage config property validation.
*/
@Beta
public class InvalidStagePropertyError extends ValidationError {
/**
* Creates validation error that occurred during stage validation.
* @param message error message
* @param stage name of the stage that caused this validation error
* @param property property that is invalid
*/
public InvalidStageError(String message, String stage, String property) {
super(message);
Map<String, Object> map = new HashMap<>();
map.put("stage", stage);
map.put("property", property);
causes.add(map);
}
} |
Code Block |
---|
/**
* Represents invalid input schema error.
*/
public class InvalidInputSchemaError extends NewValidationError {
/**
* Creates invalid input schema error.
* @param message error message
* @param stage name of the stage
* @param map map of incoming stage name to field that is invalid.
*/
public InvalidInputSchemaError(String message, String stage, Map<String, String> map) {
super(message);
for (Map.Entry<String, String> entry : map.entrySet()) {
Map<String, Object> causeMap = new HashMap<>();
causeMap.put("stage", stage);
causeMap.put("input_stage", entry.getKey());
causeMap.put("field", entry.getValue());
causes.add(causeMap);
}
}
} |
Code Block |
---|
language | java |
---|
title | InvalidOutputSchemaError.java |
---|
|
/**
* Represents invalid output schema error.
*/
public class InvalidOutputSchemaError extends NewValidationError {
/**
* Creates invalid output schema error.
* @param message error message
* @param stage name of the stage
* @param map map of output going port name to field that is invalid
*/
public InvalidOutputSchemaError(String message, String stage, Map<String, String> map) {
super(message);
for (Map.Entry<String, String> entry : map.entrySet()) {
Map<String, Object> causeMap = new HashMap<>();
causeMap.put("stage", stage);
causeMap.put("output_port", entry.getKey());
causeMap.put("field", entry.getValue());
causes.add(causeMap);
}
}
} |
API usage in plugins
The above api usage in the plugins is same as approach-1.
Code Block |
---|
@Override
public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
pipelineConfigurer.createDataset(conf.destinationFileset, FileSet.class);
StageConfigurer stageConfigurer = pipelineConfigurer.getStageConfigurer();
// get the name of the stage
String stageName = stageConfigurer.getStageName();
try {
Pattern.compile(conf.filterRegex);
} catch (Exception e) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStagePropertyError(e.getMessage(), stageName, "filterRegex"));
}
if (conf.sourceFileset.equals(conf.destinationFileset)) {
// add validation error to stage configurer
stageConfigurer.addValidationError(new InvalidStageError("source and destination filesets must be different", stageName));
}
} |
Impact on UI
Type | Description | Scenario | Approach - 1 - Json Response | Approach - 2 - Json Response |
---|
STAGE_ERROR | Represents validation error while configuring the stage | If there is any error while connecting to sink while getting actual schema | { "errors": [ { "type" : "STAGE_ERROR",
"stage" : "src",
"message" : "source and destination filesets must be different" } ] } | { "errors": [ { "message": "source and destination filesets must be different", "causes": [ { "stage": "src" } ] } ] } |
INVALID_PROPERTY | Represents invalid configuration property | If config property value contains characters that are not allowed by underlying source or sink | { "errors": [ { "type" : "INVALID_PROPERTY",
"stage" : "src",
"message" : "Invalid config for property 'port'", "property" : "port" } ] } | { "errors": [ { "message": "Invalid config for property 'port'", "causes": [ { "stage": "src", "property": "port" } ] } ] } |
PLUGIN_NOT_FOUND | Represents plugin not found error for a stage | If the plugin was not found. This error will be thrown from the data pipeline app | { "errors": [ { "stage": "src", "type": "PLUGIN_NOT_FOUND", "message": "Plugin named 'Mock' of type 'batchsource' not found.", "pluginType": "batchsource", "pluginName": "Mock", "requestedArtifact": { "scope": "USER", "name": "app-mocks-ghost", "version": "1.0.0" } } ] } | { "errors": [ { "message": "Plugin named 'Mock' of type 'batchsource' not found.", "causes": [ { "stage": "src", "pluginType": "batchsource", "pluginName": "Mock", "requestedArtifact": { "scope": "USER", "name": "app-mocks-ghost", "version": "1.0.0" } } ] } ] } |
INVALID_INPUT_SCHEMA | Represents invalid schema field in input schema | If the input schema for the plugin is not supported by underlying plugin | { "errors": [ { "type" : "INVALID_INPUT_SCHEMA",
"stage" : "joiner",
"message" : "Invalid schema field 'id'. It should be of type 'string'", "field" : "id" } ] } | {
"errors": [ { "message": "Invalid schema field 'id'. It should be of type 'string'", "causes": [ { "stage": "joiner", "input_stage": "source1", "field": "id" }, { "stage" : "joiner", "input_stage" : "source2", "field" : "id" } ] } ] } |
INVALID_OUTPUT_SCHEMA | Represents invalid schema field in output schema | If the output schema for the plugin is not compatible with underlying sink | { "errors": [ { "type" : "INVALID_OUTPUT_SCHEMA",
"stage" : "splitter",
"message" : "Invalid schema field 'email'. It should be of type 'string' at port 'port'", "field" : "email" } ] } | { "errors": [ { "message": "Invalid schema field 'email'. It should be of type 'string'", "causes": [ { "stage": "splitter", "output_port": "port", "field": "email" } ] } ] } |
Conclusion
Approach 1
Approach 2 is more flexible than approach 1 in terms of what error classes can have as attributes.
Jira Legacy |
---|
server | Cask Community Issue Tracker |
---|
serverId | 45b48dee-c8d6-34f0-9990-e6367dc2fe4b |
---|
key | CDAP-15578 |
---|
|
Future Work
- Add pipeline level validation such as DAG validation
Test Scenarios
Test ID | Test Description | Expected Results |
---|
|
|
|
|
|
|
|
|
|
|
|
|
Releases
Release 6.1.0
/**
*
* @param message
* @param stage
*/
public NewStageValidationError(String message, String stage) { super(message);
causes.put("stage", stage);
}