Motivation
We are trying to improve the user experience for Hydrator Plugins. As a first step we will like to achieve the following:
- Catch pipeline misconfiguration during deployment
- Dataset checks during deployment
- Improve error messages on failure
- Fail early, fail fast.
To do this we are trying to collect all the known pipelines failures and other annoyances. This can be anything from failures due to non-existing sources to missing tooltips. We are going through all the JIRA, support tickets and user groups to collect this information and will appreciate your help in it. If you are aware of issues in Hydrator pipelines which can improve the usability and user experience please report it in the table below.
| Issue | Affected Source/Sink/Transform | Caused by (if known) | JIRA (if existing) | Error message |
|---|---|---|---|---|
| Importing a pipeline needs users to click on nodes to populate schema | ALL | Error rendering macro 'jira' : Unable to locate Jira server for this macro. It may be due to Application Link configuration. | ||
| Getschema for DB plugins should honor the column case configuration | DB Plugins | |||
| Error messages from plugins should have the plugin name, version and scope | Error messages from plugins should have the plugin name, version and scope | |||
| Schema requirement of plugins is not visible to users | java.util.concurrent.ExecutionException: java.lang.RuntimeException: Exception while configuring plugin of type batchsink and name Table for stage Table: Field type mismatch, field 'output' type in input schema is ["string","null"], while in output schema its of type "string" | |||
| If a plugin throws an exception, the pipeline fails without logging the error | ||||
| Hydrator pipeline fails at runtime when external sink has the same name as native sink | Pipeline with TPFS sink (called output) and HDFS sink with external name called output | java.lang.IllegalArgumentException: An external sink cannot have the same name as an existing CDAP Dataset instance output | ||