Introduction 

Error handling in CDAP is important for better user experience. One of the challenge is to standardize error messages CDAP Data pipeline and Wrangler 

Goals

There are four goals which needs to be achieved to improve error handling:

Introduction 

Error handling in CDAP is important for better user experience. 

Goals

There are four goals which needs to be achieved to improve error handling:

• Add a framework to centralize error messages in wrangler and

data

• pipeline

to provide error codes along with error messages for better user experience.

Scenarios

• Scenario 1: Error codes in Wrangler

  • Scenario 1.1

Alice wants to wrangle data using CDAP's Wrangler tool. As part of that, Alice wants to connect to Database Source using Wrangler Connection. While attempting to do that, Alice is seeing a cryptic error message while testing the connection that she does not know how to resolve just by looking at the error message. In order figure out cause of the issue and recommended action, Alice will like to browse error code catalog with the error code displayed along with the error message.

• • Scenario 1.2

Alice is applying transformations to the data on the fly using CDAP's Wrangler tool. However, while applying transformations, Alice sees an error message that does not suggest recommended action to fix the issue. She wants to browse the error code catalog to figure out recommended action to resolve the issue.

• • Scenario 1.3

Alice wants to wrangle data using CDAP's Wrangler tool. While applying transformations to the connected source data on the fly, Alice tries to parse boolean column as csv. However, Alice observes that built-in directives to parse boolean data as csv and avro returns different error messages. Alice would like to see standard error message from both the directives when column type is incompatible.

• • Scenario 1.4

Alice wants to wrangle data using CDAP's Wrangler tool. While connecting to the source Alice uses user name and password stored in secure store. However, the secure store keys are not resolved. Alice wants to look at the error code and share it with the CDAP developer to figure out the underlying root cause.

• Scenario 2: Error codes in Pipeline

  • Scenario 2.1

Alice is a pipeline developer who has built a pipeline using CDAP data pipeline studio. While running the pipeline, it failed with no meaningful error message in logs.To debug this issue further, Alice wants to share error code from the logs with Bob, the data pipeline app developer to figure out what is the cause of the error.

• • Scenario 2.2

As a data pipeline developer, Alice is trying to get schema from the database source. While attempting to do that from CDAP data pipeline studio, Alice sees an error message. In order to figure out the cause and action for the error message, Alice will like to search the error code in the error catalog.

• • Scenario 2.3

As a data pipeline developer, Alice is trying to validate the pipeline. While attempting to validate it, Alice saw an error message that requires fixing the plugin property. Alice want to have an error code along with the error message so that she can search the probable cause on error catalog.

• • Scenario 2.4

As a data pipeline developer, Alice is using secure macros in the pipeline, however, the secure macros are not resolved. Alice wants to use the error code and share it with CDAP developer to figure out the root cause.

• Scenario 3: Error codes in CDAP

  • Scenario 3.1

As a CDAP user, Alice wants to upload a plugin to CDAP. However, Alice is not deploy a plugin and gets an error. Alice wants to use the error code to browse the catalog to figure out the cause and recommended action.

• • Scenario 3.2

As a CDAP user, Alice wants to enable Wrangler to wrangle data. However, Wrangler service is not starting up.Alice wants to use the error code so that it can be shared with the support engineer for further debugging.

• • Scenario 3.3

As a CDAP user, Alice wants to enable Wrangler to wrangle data. However, Wrangler service is not starting up.Alice wants to use the error code so that it can be shared with CDAP developer to figure out the root cause.

Design

Approach

Impact on UI

Test Scenarios

Test ID

Test Description

Expected Results

Bug Fixes 

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-14378

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15499

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15507

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15040

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15593

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-14797

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15563

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15560

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-11767

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15426

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15581

Releases

Release 6.1.0

Related Work

• Pipeline System Application

Future work

• • Add a framework to add error codes along with error messages so that developers can figure out the source of error message without looking at stacktrace

  Add a framework to centralize error messages in wrangle

• Add a framework to add error codes to error messages so that developers can figure out the source of error message without looking at stacktrace

User Stories

1. As a CDAP pipeline developer, if a pipeline contains plugin configurations which are invalid, I will like it to fail early with appropriate error message.

2. As an ETL engineer, if I run into error situation while applying directives, I will like to see appropriate error message which clearly indicates the error.

Scenarios

• Scenario 1: Error codes in Wrangler

  • Scenario 1.1

Alice wants to wrangle data using CDAP's Wrangler tool. As part of that, Alice wants to connect to Database Source using Wrangler Connection. While attempting to do that, Alice is seeing a cryptic error message while testing the connection that she does not know how to resolve just by looking at the error message. In order figure out cause of the issue and recommended action, Alice will like to browse error code catalog with the error code displayed along with the error message.

• • Scenario 1.2

Alice is applying transformations to the data on the fly using CDAP's Wrangler tool. However, while applying transformations, Alice sees an error message that does not suggest recommended action to fix the issue. She wants to browse the error code catalog to figure out recommended action to resolve the issue.

•  Scenario Scenario 2: Centralized Error messages in Wrangler

• • Scenario 2.1

Alice wants to wrangle data using CDAP's Wrangler tool. While applying transformations to the connected source data on the fly, Alice tries to parse boolean column as csv. However, Alice observes that built-in directives to parse boolean data as csv and avro returns different error messages. Alice would like to see standard error message from both the directives when column type is incompatible.

• Scenario 3: Error codes in Pipeline

  • Scenario 3.1

As a plugin developer, Bob has deployed a plugin to CDAP. Alice is a pipeline developer who has built is building a pipeline using CDAP data pipeline studio. While running the pipeline, it building the pipeline,  built data pipeline using GUI of CDAP pipeline studio. After deploying the data pipeline, the pipeline failed with NPE without any meaningful error message. To debug this issue further, Alice wants to share error code from the logs , rather than whole stack trace, with Bob to figure out what is the cause of the error.

• • Scenario 3.2

As a data pipeline developer, Alice is trying to get schema from the BigQuery source. While attempting to do that from CDAP data pipeline studio, Alice saw an error message 'Name is null' that does not provide information about the underlying issue. Alice want to have an error code along with the error message so that she can search the probable cause on error catalog.

• • Scenario 3.3

As a data pipeline developer, Alice is trying to validate the pipeline. While attempting to validate it, Alice saw an error message due to the bug in the plugin implementation. Alice would like to us error code to browse the catalog`that does not provide information about the underlying issue. Alice want to have an error code along with the error message so that she can search the probable cause on error catalogwhich method caused NPE exception.

• Scenario 4: Centralized Error messages in Pipeline

  • Scenario 4.1

Alice who is a data pipeline developer, develops data pipelines using CDAP. While building data pipelines, Alice observes that the error messages for common error types such as invalid configuration error are not consistent. Alice would like to see consistent error messages for common errors.

Design

Approach

Impact on UI

Test Scenarios

Bug Fixes 

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-14378

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15499

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15507

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15040

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15593

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-14797

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15563

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15560

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-11767

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15426

• Jira Legacy
  server Cask Community Issue Tracker columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 45b48dee-c8d6-34f0-9990-e6367dc2fe4b key CDAP-15581

Releases

Release 6.1.0

Related Work

• Pipeline System Application

Future work