...
| Code Block |
|---|
{
"stages": [
{
"name": "customers",
"plugin": {
"name": "File",
"type": "batchsource",
"properties": {
"path": "hdfs://host:port/${inputpath}" // ${inputpath} will get replaced with the value of the 'customers.inputpath' runtime argument
}
}
},
{
"name": "items",
"plugin": {
"name": "File",
"type": "batchsource",
"properties": {
"path": "hdfs://host:port/${inputpath}" // ${inputpath} will get replaced with the value of the 'items.inputpath' runtime argument
}
}
}
]
} |
Changes to Existing Plugins (WIP)
Many plugins have fields (configurable properties) that are used in constructing a schema at configure time. These fields need to have macros disabled. The following plugins would be affected:
| Plugin | Fields | Notes |
|---|---|---|
| BatchCassandraSource | schema | The schema is parsed for correctness. |
| RealtimeCassandraSink | addresses | Addresses are parsed at configure time. Parsing a macro would fail. |
| CopybookSource | copybookContents | Copybook contents are converted to an InputStream and used to get external records, which are in turn used to add fields to the schema. |
| DedupAggregator | uniqueFields, filterOperation | Both fields are used to validate the input schema created |
Setting Hydrator runtime arguments using CDAP runtime arguments/preferences
...