Macro Substitution

Previously, the configurations of plugins in an ETL pipeline could not be changed after the pipeline's deployment. Macro substitution provides pipeline developers and operators the ability to configure plugin settings on a run-to-run basis for settings that may be unknown at the time of configuration.

Specifying Substitutions

Macro substitution provides two types of substitutions through property lookups and macro functions. Property lookups are specified through key-value pairs. There are two ways to specify these key-value pairs.

1. Set a key-value pair in the runtime arguments or preferences for the physical pipeline.
2. Custom Actions can be run in the first stage of a pipeline and set a key-value argument through a workflow token.

In order to enable a plugin property as macro-substitutable, the property must be both annotated as macro-enabled and provided proper macro syntax at configuration time.

Plugin Config Annotation

In order to enable macro substitution for a property, use the @Macro annotation on the property field in a plugin's configuration class. As such, macros are disabled for all fields by default. For example:

public class TableSinkConfig extends PluginConfig {
  @Name(Properties.Table.NAME)
  @Description("Name of the table. If the table does not already exist, one will be created.")
  // The name of the table can be specified by a runtime macro
  @Macro 
  private String name;

  @Name(Properties.Table.PROPERTY_SCHEMA)
  @Description("schema of the table as a JSON Object. If the table does not already exist, one will be " +
    "created with this schema, which will allow the table to be explored through Hive. If no schema is given, the " +
    "table created will not be explorable.")
  @Nullable
  private String schemaStr;

  @Name(Properties.Table.PROPERTY_SCHEMA_ROW_FIELD)
  @Description("The name of the record field that should be used as the row key when writing to the table.")
  private String rowField;
}

Syntax

In addition to a plugin property being annotated with @Macro, proper macro syntax must be provided to the property field at configure time. There are two valid macro syntaxes, property lookups and macro functions.

Property Lookup

Macro property lookups are simple key-value substitutions that use the following syntax:

${macro-name}

At runtime, the syntax ${macro-name} will be replaced with whatever value was specified for the key "macro-name." For instance, you might not know the name of a source stream until runtime. You could use, in the source stream's Stream Name configuration:

${source-stream-name}

and in the runtime arguments (or preferences) set a key-value pair such as:

source-stream-name: myDemoStream

Macros can be referential. You might have a server that refers to a hostname and port, and specify this substitution:

server-address: ${hostname}:${port}

and these runtime arguments:

hostname: my-demo-host.example.com
port: 9991

In a pipeline configuration, you could configure a property with:

${server-address}

expecting that it would be replaced with:

my-demo-host.example.com:9991

Lookup Precedence

For property lookup macros, substitutions set through workflow tokens take precedence over those set through runtime arguments and preferences.

Macro Function

Macro functions allow more complex logic to be run before a substitution occurs and use the following syntax:

${macroFunction(arg1,arg2,arg3)}

At runtime, the "macroFunction" function will perform some computation with the provided arguments: arg1, arg2, and arg3. Note that whitespace is significant in arguments. The syntax will be replaced with whatever macroFunction evaluates to given the provided arguments.

Currently, there are two support macro functions, logicalStartTime and secure.

logicalStartTime

${logicalStartTime(timeFormat,offset)}

The logicalStartTime macro function takes in a time format and an optional offset as arguments and uses the logical start time of a pipeline to perform the substitution. For example, suppose the logical start time of a pipeline run is 2016-01-01T00:00:00 and the following macro is provided:

${logicalStartTime(yyyy-MM-dd'T'HH-mm-ss,1d-4h+30m)}

The format is yyyy-MM-dd'T'HH-mm-ss and the offset is 1d-4h+30m before the logical start time. This means the macro will be replaced with 2015-12-31T03:30:00, since the offset translates to 20.5 hours. Therefore, the entire macro evaluates to 20.5 hours before midnight of new years 2016.

secure

${secure(key)}

The secure macro function takes in a single key as an argument and looks up the key's associated string value from the Secure Store. In order to perform the substitution, the key provided as an argument must already exist in the secure store. This is useful for performing a substitution with sensitive data.

For example, for a plugin that connects to a MySQL database, you can configure the "password" property field with:

${secure(mysql-password)}

which will pull "mysql-password" from the secure store at runtime.

Expansion Capabilities

Macro parsing supports the following capabilities:

Multiple Macros

A single property can have multiple macros either consecutive or spread out.

${host}/${path}:${port} --> example.com/index.html:80

Nested Macros

A macro can have another macro nested inside. Nested macros are expanded before the outer macro. If there are multiple nested macros within a single enclosing macro, they are evaluated from right-to-left. Based on the specified substitutions, the following substitutions could be possible based on the value of "host-suffix":

Option 1. ${hostname${host-suffix}} --> ${hostname-use-suffix} --> example.com/index.html:80
Option 2. ${hostname${host-suffix}} --> ${hostname-dont-use-suffix} --> example.com

Recursive Macros

A macro can be substituted with another macro if specified as such. For example, the following key-value pair:

full-server-address: ${hostname}/${path}:${port}

would lead to the following expansion:

${full-server-address} --> ${hostname}/${path}:${port} --> example.com/index.html:80

In the case that a single macro expands to multiple macros, the new macros will be substituted from right-to-left. Recursive expansion is currently supported up to a maximum depth of 10.

Macro Syntax Escaping

Macro syntax can be escaped. To escape existing syntax, use the backslash '\' character. Backslashes themselves must also be escaped, so to use a backslash, use two consecutive backslashes '\\'. For example, the following syntax:

${\${escaped-macro-literal\}}

would be look up the specified property with a key of: ${escaped-macro-literal}.