...
| Plugin | Fields | Use | Conflict |
|---|---|---|---|
| BatchCassandraSource | schema | Parsed for correctness to create the schema. | Parsing a macro or schema with a nested macro would fail. |
| CopybookSource | copybookContents | Copybook contents are converted to an InputStream and used to get external records, which are in turn used to add fields to the schema. | Schema would add macro literal as a field. |
| DedupAggregator | uniqueFields, filterOperation | Both fields are used to validate the input schema created. | Macro literals do not exist as fields in schema and will throw IllegalArgumentException. |
| DistinctAggregator | fields | Used to specify Specifies the fields used to construct the output schema. | Will add macro literals as schema fields.* |
| GroupByAggregator | groupByFields, aggregates, | Gets fields from input schema and adds aggregates to to output fields list. | Macro literals do not exist in input schema or are valid fields for an output schema. |
| RowDenormalizerAggregator | keyField, nameField, valueField | Gets schemas by field names from the input schema. | Macro literals do not exist as fields in the input schema. |
| BatchWritableSink | name, type | Creates dataset from properties. | Macro literals will lead to a misnamed dataset and type. |
| KVTableSink | keyField, valueField | Validates that presence and type of these fields in the input schema. | Macro literals will not exist in the input schema. |
| ReferenceBatchSink | referenceName | Verifies reference name meets dataset ID constraints. | Macro literal may not meet constraints. |
| SnapshotFileBatchAvroSink | schema | Parses schema to add file properties. | Macro literals may disallow schema parsing or incorrect schema creation. |
| SnapshotFileBatchParquetSink | schema | Parses schema to add file properties. | Macro literals may disallow schema parsing or incorrect schema creation. |
| SnapshotFileBatchSink | name | Creates dataset from name field. | Name of data set will change on macro substitution. |
* May need verification
Other plugins have fields that are validated/processed at configure time that do not affect the schema. In these cases, these can be moved to initialization time. The following plugins and fields would be affected:
...