...
| Code Block |
|---|
Example for Scoping: Provided runtime arguments: Key : table-name, value : employees Key : TableSink:table-name, value : employee_sql table-name is the macro name that is used in both DBSource stage and TableSink stage. if user wants to provide a special value for macro "table-name" to be used in TableSink, he will prefix stage-name before the macro name separated by the delimiter (colon). |
Thoughts from Terence:
Below are the thoughts I have so far.
1. Preferences/runtime arguments substitution for configuration values
- Can start with simple $var substitution
- The DataPipeline app performs the substitution
- The perferences can be scoped
- Properties prefixed with the plugin name (stage name?) will be striped
- Property in more specific scope will override the less specific one
- e.g. If having both "password" => "a" and "plugin1.password" => "b" in perferences, then for Plugin "plugin1", it will see "password" => "b"
- For managing passphase so that plugin config will only contains key name, but not the actual key
- Plugins that need sensitive information need to be adjusted to use the key management
- Potentially can have the DataPipeline app do the substitution as well
- But we cannot use "$", since it's used above. Maybe can be "#".
- E.g. for plugin config {"password" => "#dbpassword"}, then at runtime the actual password with name "dbpassword" will be fetched from the secure store.
Reference:
Changes to Existing Plugins
...