...
- User stories documented (Albert/Vinisha)
- User stories reviewed (Nitin)
- Design documented (AlbertShankar/VinishaKashif)
- Design reviewed (Terence/Andreas)
- Feature merged ()
- Examples and guides ()
- Integration tests ()
- Documentation for feature ()
- Blog post
...
PluginConfigurer Changes:
PluginConfigurer will expose the following field
| Code Block |
|---|
public interface PluginConfigurer extends DatasetConfigurer {
/**
* If the plugin field can accept macro and if the config for the plugin has a macro, then return true, else return false.
*/
boolean isMacro(String fieldName);
} |
The method will return whether or not the property with the provided fieldName contained contains a macro at configure time. We don't want to force deferring macros to runtime in the case that a field is macroable but actually has no macro provided in its configuration. This allows optional checking of properties at configure time for simple pipelines.
| Code Block | ||
|---|---|---|
| ||
@Override
void configurePipeline(PipelineConfigurer pipelineConfigurer) {
if (!pipelineConfigurer.isMacro("datasetName")) {
// create dataset if the datasetName field is not a macro
pipelineConfigurer.createDataset(datasetName, datasetType, DatasetProperties.builder().addAll(properties).build());
}
...
} |
...
- To allow macro substitution of non-String properties, any properties configured with a macro will have a placeholder default value at configure time which is different from runtime.
- For primitive types, this would be Java's default value, Example: integer field will have a value of 0. objects, would be null.
- This is fine because we are exposing a method to check whether a field is a macro or not in pipelineConfigurer. so plugin developers can check at configure time if a field is macro before performing appropriate action using that field.
- During runtime, macro fields will be substituted and plugin instance will have fields with substituted values.
PluginContext Changes:
| Code Block |
|---|
@Beta
public interface PluginContext {
/**
* Creates a new instance of a plugin. The instance returned will have the {@link PluginConfig} setup with
* {@link PluginProperties} provided at the time when the
* {@link PluginConfigurer#usePlugin(String, String, String, PluginProperties)} was called during the
* program configuration time. If a plugin field has a macro,
* the parameter evaluator is used to evaluate the macro and the evaluated value is used for the plugin field.
*
* @param pluginId the unique identifier provide when declaring plugin usage in the program.
* @param evaluator the macro evaluator that's used to evaluate macro for plugin field if macro is supported on those fields.
* @param <T> the class type of the plugin
* @return A new instance of the plugin being specified by the arguments
*
* @throws InstantiationException if failed create a new instance
* @throws IllegalArgumentException if pluginId is not found
* @throws UnsupportedOperationException if the program does not support plugin
*/
<T> T newPluginInstance(String pluginId, MacroEvaluator evaluator) throws InstantiationException; |
| Code Block | ||
|---|---|---|
| ||
@Beta
interface MacroEvaluator {
/**
* forUse the macro function and call the function with provided arguments,
* apply appropriate function or use properties in default case and return the evaluated value
*/
String evaluate(String macroFunction, String... arguments);
} |
Notes:
- During configure time in the configurePipeline method, if a field is macro enabled, the property should not be validated as the macro has not been provided a substitutable value.
- During runtime, PluginInstantiator has to understand if macro is enabled for a field, it parses the macro and then uses the parameter `evaluator` to evaluate the macro.
- The MacroEvaluator is provided by the DataPipelineApp. Though the platform understands macro, MacroEvaluator enables the actual substitution logic to be provided by the Application.
...