Table of Contents |
---|
Checklist
- User Stories Documented
- User Stories Reviewed
- Design Reviewed
- APIs reviewed
- Release priorities assigned
- Test cases reviewed
- Blog post
Introduction
Enhancements to wrangler. Ability to read connect to external sources like Database, Kafka and perform wrangler directives on them for data preparation.
Goals
Ability to dynamically load plugins from CDAP Service.
User Stories
- User wants to load the database driver artifact in wranger and execute database commands to load data. Once data is loaded they want to execute wrangler directives on the loaded data.
- User adds kafka artifact, immediately wants to load the kafka artifact in wrangler and read data from a kafka topic, once data is loaded, they want to execute wrangler directives on the loaded data.
- User Story #3
Design
Background :
Cover details on assumptions made, design alternatives considered, high level design
Approach
Approach #1
Approach #2
API changes
New Programmatic APIs
New Java APIs introduced (both user facing and internal)
Code Block | ||||
---|---|---|---|---|
| ||||
public interface PluginContextAdmin { ... /** * Gets Readonly Artifact Repository, ReadOnlyArtifactRepository provides ability to get artifacts by plugin type and name. Get plugin class * for a given artifactId, pluginType and PluginName and also instantiate this plugin. * You can only perform read only operation on artifacts and instantiate plugins using the artifact. However any delete or modify operation * is now allowed. * Since this provides live view of Artifact Repository, same call could provide different results, if there is any change in repository. * * * * @return A read only access to artifact repository * */ ReadOnlyArtifactRepository getArtifactRepository(); ... }extends DatasetManager, SecureStoreManager, MessagingAdmin, ArtifactManager { } |
Code Block |
---|
/** * ReadProvides only access to artifact repository.information about artifacts */ classinterface ReadOnlyArtifactRepositoryArtifactManager { /** * forreturns the pluginlist typeof andartifacts pluginavailable name,in getcurrent allnamespace theand artifact parents that are usableBy the pluginsystem namespace. */ @param pluginTypeList<ArtifactDetail> listArtifacts(); /** @param pluginName * @returnCreate setand ofreturn parentclass artifactsloader whichwith extends the plugincontents identified by pluginType and pluginName. */ public Set<ArtifactId> getParentArtifactsForPlugin(String pluginType, String pluginName) /** * Get the Plugin identified by parent artifact Id, pluginType and pluginName * @param parentArtifactId * @param pluginType * @param pluginName * @return Plugin * throws PluginNotExistsException if the plugin is not found */ public Plugin getPlugin(ArtifactId parentArtifactId, String pluginType, String pluginName) /** * Instantiate the supplied plugin using the provided ClassLoader and return the instantiated plugin. * @param classLoader * @param loaded plugin * @return instantiated plugin * throws InstantiationException if the plugin cannot be instantiated. */ public <T> T instantiatePlugin(ClassLoader classLoader, Plugin plugin) } of artifactDetail, with the parentClassLoader as parent, * if parentClassLoader is null we will use boot strap class loader as the parent. * @return ClassLoader */ ClassLoader createClassLoader(ArtifactDetail artifactDetail, @Nullable ClassLoader parentClassLoader); ... } |
Notes :
1) ArtifactId class is available in cdap-api, we might want to move ArtifactDetail or create another object in cdap-api exposing artifact details, With Admin, CDAP developers can access the ArtifactManager and get information about artifacts
2) How the user can instantiate the plugin ?
Since InstantatiorFactory is available as part of cdap-common, Once they have loaded the desired plugin class, they can use the instantiatorFactory to instantiate the plugin.
Deprecated Programmatic APIs
New REST APIs
Path | Method | Description | Response Code | Response |
---|---|---|---|---|
/v3/apps/<app-id> | GET | Returns the application spec for a given application | 200 - On success 404 - When application is not available 500 - Any internal errors |
|
Deprecated REST API
Path | Method | Description |
---|---|---|
/v3/apps/<app-id> | GET | Returns the application spec for a given application |
CLI Impact or Changes
- Impact #1
- Impact #2
- Impact #3
UI Impact or Changes
- Impact #1
- Impact #2
- Impact #3
Security Impact
What's the impact on Authorization and how does the design take care of this aspect
Impact on Infrastructure Outages
System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect
Test Scenarios
Test ID | Test Description | Expected Results |
---|---|---|
Releases
Release X.Y.Z
Release X.Y.Z
Related Work
- Work #1
- Work #2
- Work #3