...
1. Preferences/runtime arguments substitution for configuration values
- Can start with simple $var substitution
- The DataPipeline app performs the substitution
- The perferences can be scoped
- Properties prefixed with the plugin name (stage name?) will be striped
- Property in more specific scope will override the less specific one
- e.g. If having both "password" => "a" and "plugin1.password" => "b" in perferences, then for Plugin "plugin1", it will see "password" => "b"
- For managing passphase so that plugin config will only contains key name, but not the actual key
- Plugins that need sensitive information need to be adjusted to use the key management
- Potentially can have the DataPipeline app do the substitution as well
- But we cannot use "$", since it's used above. Maybe can be "#".
- E.g. for plugin config {"password" => "#dbpassword"}, then at runtime the actual password with name "dbpassword" will be fetched from the secure store.
----------------------
Setting Hydrator runtime arguments using CDAP runtime arguments/preferences
CDAP preferences and runtime arguments will be used directly as Hydrator arguments.
1.) Runtime arguments can be passed to hydrator pipeline in 2 ways:
- Using Prepipeline-CustomActions:
Prepipeline custom actions can set runtime arguments. For example, before running the pipeline, custom actions can copy local files to hdfs and set runtime arguments for input path for batchsource. In order to do that, we can expose setPreferences() and getPreferences() programmatic api for setting runtime arguments. These arguments can be passed to hydrator app using workflow token. - Using Hydrator UI:
For each stage, runtime arguments can be passed from hydrator UI using cdap REST endpoints for preferences/runtime arguments framework.
2.) Hydrator app will substitute properties using Macro substitution for each ETLStage. Now, plugins, like SFTP, which need secure substitution using key management can use 'secure' prefix in the macro. Macro substitution should vary depending on prefix of the arguments. In case of secure key, macro can be '${secure(key)}', in case of value directly to be substituted, macro can be '${inputpath}' without any prefix.
----------------------------
Reference:
Changes to Existing Plugins
...