Goals

Checklist

•  User stories documented (Albert/Vinisha) 
•  User stories reviewed (Nitin)
•  Design documented (Shankar/Kashif)
•  Design reviewed (Terence/Andreas)
•  Feature merged ()
•  Examples and guides ()
•  Integration tests () 
•  Documentation for feature ()
•  Blog post

...

Function Syntax : 
${macroFunction(macro)}
 
ShorthandProperty lookup notationsyntax:
${macro}
 
Example Usage: 
${secure(accessKey)} - get access key from secure store 
${logicalStartTime(timeFormat)} - apply time function on the timeFormat provided and use the value. 
 
The Default (shortHand) usage will substitute arguments using the following precedence:
Custom Action Workflow-Token > Runtime Arguments > Stored Preferences
 
Examples:
ipConfig: ${hostname}:${port}
JDBC connection string : jdbc:${jdbc-plugin}://${hostname}:${sql-port}/${db-name}
 
Using the expanded syntax allows additional logic to be applied to the macro arguments through a macro function.
Escaping can be supported using the \ (backslash) character (e.g. \${hostname} will not be substituted)
 
Nested macros:
if a macro contains another macro, Example : ${secure(${user-name})} 
In the above example, we want to lookup the user-name in properties first, then use secure store to get the key/password for that user-name. this final key/password will be used for that field. 

The shorthand notation supports retrieval precedence to limit the exposure of underlying workflow-tokens and runtime-arguments to pipeline operators. The "functionTime" macro function uses the logical start time of a run to perform the substitution. This is an example of a macro function that is not just a key-value lookup but allows for extra logic to be performed before a value is returned. For now, the implementation will only support the following macro functions: runtime-arguments. Once the secure store API is available, it will also support secure store. In the future, we can see if we will allow developers to create custom macro functions (similar to functionTime(...)).

Notes:

• For now, we will not support The current implementation for macro substitution supports recursive expansion of macros. That is, if a macro such as ${address} expands to ${hostname}:${port}, then ${hostname} and ${port} will not be evaluated. We will document this at first.We can expand the functionality later to recursively expand macros.In the case of a macro ${key} expanding to ${key}, we can implement a maximum depth of recursionHowever, this can lead to an infinite loop from circular macros, so we can add a maximum depth for expansion.

"stages": [
    {
        "name": "Database",
        "plugin": {
            "name": "Database",
            "type": "batchsource",
            "properties": {
				...
                "user": "${username}",
                "password": "${secure(sql-password)}",
                "jdbcPluginName": "jdbc",
                "jdbcPluginType": "${jdbc-type}",
                "connectionString": "jdbc:${jdbc-type}//${hostname}:${port}/${db-name}",
                "importQuery": "select * from ${table-name};"
            }
        }
    },
    {
        "name": "Table",
        "plugin": {
            "name": "Table",
            "type": "batchsink",                                        
            "properties": {
                "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",
                \"fields\":[{\"name\":\"name\",\"type\":\"string\"},
                {\"name\":\"age\",\"type\":\"int\"},{\"name\":\"emp_id\",\"type\":\"long\"}]}",
                "name": "${table-name}",
                "schema.row.field": "name"
            }
        }
    }
]

...