...
| Code Block | ||
|---|---|---|
| ||
interface MacroContext {
/**
* Given the macro key, return the translatedsubstituted value
*/
String getValue(String macroKey);
} |
...
| Code Block | ||
|---|---|---|
| ||
Based on the macro type, one of the below MacroContext's will be used to get the value for macro.
DefaultMacroContext implements MacroContext {
Map<String, String> runtimeArguments;
String getValue(String macroKey) {
return runtimeArguments.get(macroKey);
}
}
SecureMacroContext implements MacroContext {
SecureStore secureStore;
String getValue(String macroKey) {
return secureStore.get(macroKey);
}
}
RuntimeFunctionMacro implements MacroContext {
TimeZone timeZone;
long logicalStartTime;
Function<String, String> timezoneFunction;
String getValue(String arguments) {
return timezoneFunction.apply(arguments);
}
}
|
Scoping:
Since the macro-substitution is performed at the DataPipeline app level, it will be possible to scope at stage name level if the user desires that.
In our example config of JDBC source to Table sink, there is a common macro "${table-name}", if the user wants to provide a different name for the table-name in Table Sink, he can use scoping.
| Code Block |
|---|
Format : stage_name:key
Example for Scoping:
Key : table-name value : employees // non-scoped key and value
Key : tableSink.table-name value : employee_sql // scoped key and value
//Priority will be provided for scoped key with stage-name if that key is present, else non-scoped key will be used if that is present.
String substituteAndGet(String macro, String stageName) {
if (runtimeArgs.containsKey("<stagename>.<macro>")) {
return substituteMacro(macroContext.get(<stagename>.<macro>))
} else if (runtimeArgs.containsKey(<macro>)) {
return substituteMacro(macroContext.get(<macro>));
} else {
throw MacroNotFoundException("Expected macro %s is not found", <macro>);
}
} |
Hydrator Plugin Changes
Currently when we deploy a pipeline, configurePipeline is called on each plugin. we perform few validations in configure stage, specifically for schema, syntax for scripts, etc. In some Plugins we also create dataset if the dataset doesn't already exist.
...