Checklist
- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
Introduction
Enhancements to wrangler. Ability to read connect to external sources like Database, Kafka and perform wrangler directives on them for data preparation.
Goals
Ability to dynamically load plugins from CDAP Service.
User Stories
- User wants to load the database driver artifact in wranger and execute database commands to load data. Once data is loaded they want to execute wrangler directives on the loaded data.
- User adds kafka artifact, immediately wants to load the kafka artifact in wrangler and read data from a kafka topic, once data is loaded, they want to execute wrangler directives on the loaded data.
- User Story #3
Design
Background :
1) Currently plugins that required by programs are configured at configure time. these plugin artifacts get localized to the program containers.
2) At runtime, these plugins can be instantiated by their plugin id
3) we want to be able to dynamically instantiate plugins based on plugin type, name and version at runtime (not possible currently).
Example :
user uploads a POSTGres JDBC Driver
wants to load this POSTGres driver in wrangler,
configure connection and execute a query to select data and perform wrangler directives on the data.
Currently the wrangler service cannot load the PostGres driver, as the PostGres driver was not configured as a plugin in the service configuration. it was not localized to the container and we won't be able to instantiate it.
Cover details on assumptions made, design alternatives considered, high level design
Approach
Approach #1
Approach #2
API changes
New Programmatic APIs
New Java APIs introduced (both user facing and internal)
PluginContext.java
public interface PluginContext {
...
/**
* Creates a new instance of a plugin.
*
*
* @param pluginType plugin type name
* @param pluginName plugin name
* @param properties properties for the plugin. The same set of properties will be used to instantiate the plugin
* instance at execution time
* @param <T> the class type of the plugin
* @return A new instance of the plugin being specified by the arguments
*
* @throws InstantiationException if failed create a new instance
* @throws IllegalArgumentException if pluginId is not found
* @throws UnsupportedOperationException if the program does not support plugin
*/
<T> T newPluginInstance(String pluginType, String pluginName, PluginProperties properties) throws InstantiationException;
...
}
Deprecated Programmatic APIs
New REST APIs
| Path | Method | Description | Response Code | Response |
|---|---|---|---|---|
| /v3/apps/<app-id> | GET | Returns the application spec for a given application | 200 - On success 404 - When application is not available 500 - Any internal errors |
|
Deprecated REST API
| Path | Method | Description |
|---|---|---|
| /v3/apps/<app-id> | GET | Returns the application spec for a given application |
CLI Impact or Changes
- Impact #1
- Impact #2
- Impact #3
UI Impact or Changes
- Impact #1
- Impact #2
- Impact #3
Security Impact
What's the impact on Authorization and how does the design take care of this aspect
Impact on Infrastructure Outages
System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect
Test Scenarios
| Test ID | Test Description | Expected Results |
|---|---|---|
Releases
Release X.Y.Z
Release X.Y.Z
Related Work
- Work #1
- Work #2
- Work #3