Page Comparison

...

The dataset to write to can be macro-substituted. so we have to defer dataset creation in initialize, to prepareRun rather than doing at configure stage. 

...

// if a pipeline config fields does not support macro, can be specified at config level. 
// if not specified by default, alla fieldsfield can not be macro substituted
@Macro(enabled=falsetrue) 
public static class ValidatorConfig extends PluginConfig {
  @Description("Comma-separated list of validator plugins that are used in script")
  String validators;
  @Description(SCRIPT_DESCRIPTION)
  String validationScript;

  @Description("Lookup tables to use during transform. Currently supports KeyValueTable.")
  @Nullable
  String lookup;
}

public class TableSinkConfig extends PluginConfig {
  @Name(Properties.Table.NAME)
  @Description("Name of the table. If the table does not already exist, one will be created.")
  @Macro(enabled=true) // The name of the table can be specified by a runtime macro
  private String name;

  @Name(Properties.Table.PROPERTY_SCHEMA)
  @Description("schema of the table as a JSON Object. If the table does not already exist, one will be " +
    "created with this schema, which will allow the table to be explored through Hive. If no schema is given, the " +
    "table created will not be explorable.")
  @Nullable
  @Macro(enabled=false)	
  // The schema cannotshould not be macro substituted, otherby fieldsa canruntime be macro substituted.
  private String schemaStr;

  @Name(Properties.Table.PROPERTY_SCHEMA_ROW_FIELD)
  @Description("The name of the record field that should be used as the row key when writing to the table.")
  @Macro(enabled=true) // The name of the row field can also be specified by a runtime macro
  private String rowField;
}

...

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface Macro {

  /**
   * Default status if macro is enabled.
   */
  boolean DEFAULT_STATUS = truefalse;

  /**
   * Returns if macro is enabled. Default is 'truefalse'.
   */
  boolean enabled() default DEFAULT_STATUS;
}
 

/**
 * Contains information about a property used by a plugin.
 */
@Beta
public class PluginPropertyField {

  private final String name;
  private final String description;
  private final String type;
  private final boolean required;
  // returns true if this field can accept macro
  private final boolean macroEnabled;
  ...
}

Changes to Existing Plugins

...

Many plugins have fields (configurable properties) that are used in constructing or validating a schema at configure time. These fields need to have macros disabled to allow this. The following plugins and fields would be affected:

Plugin	Fields	Use	Conflict
BatchCassandraSource	schema	Parsed for correctness to create the schema.	Parsing a macro or schema with a nested macro would fail.
CopybookSource	copybookContents	Copybook contents are converted to an InputStream and used to get external records, which are in turn used to add fields to the schema.	Schema would add macro literal as a field.
DedupAggregator	uniqueFields, filterOperation	Both fields are used to validate the input schema created.	Macro literals do not exist as fields in schema and will throw IllegalArgumentException.
DistinctAggregator	fields	Specifies the fields used to construct the output schema.	Will add macro literals as schema fields.*
GroupByAggregator	groupByFields, aggregates,	Gets fields from input schema and adds aggregates to to output fields list.	Macro literals do not exist in input schema or are valid fields for an output schema.
RowDenormalizerAggregator	keyField, nameField, valueField	Gets schemas by field names from the input schema.	Macro literals do not exist as fields in the input schema.
KVTableSink	keyField, valueField	Validates that presence and type of these fields in the input schema.	Macro literals will not exist in the input schema.
ReferenceBatchSink	referenceName	Verifies reference name meets dataset ID constraints.	Macro literal may not meet constraints.
SnapshotFileBatchAvroSink	schema	Parses schema to add file properties.	Macro literals may disallow schema parsing or incorrect schema creation.
SnapshotFileBatchParquetSink	schema	Parses schema to add file properties.	Macro literals may disallow schema parsing or incorrect schema creation.
SnapshotFileBatchSink	name	Creates dataset from name field.	Name of data set will change on macro substitution.
TableSink	TableSink	schema, rowField	Validates output and input schemas if properties specified.	Macro literals will lead to failed validation of schema and row field.	TimePartitionedFileSetSink	all properties*	Creates dataset from properties.	Dataset will be incorrect with macro literals as fields.
TimePartitionedFileSetDatasetAvroSink	schema	Parses schema to add file properties.	Parsing macro literals in schema would fail.
TimePartitionedFileSetDatasetParquetSink	schema	Parses schema to add file properties.	Parsing macro literals in schema would fail.
ReferenceBatchSourceSnapshotFileBatchAvroSource	referenceName	Verifies that reference name meets dataset ID constraints.	Macro literals may not meet dataset ID constraints.
BatchReadableSource	name, type	Dataset is created from name and type properties.	Macro literals would lead to incorrect dataset creation.
SnapshotFileBatchAvroSource	schema	Parses schema property to set output schema.	Macro literals can lead to invalid schema schema	Parses schema property to set output schema.	Macro literals can lead to invalid schema parsing or creation.
SnapshotFileBatchParquetSource	schema	Parses schema property to set output schema.	Macro literals can lead to invalid schema parsing or creation.
SnapshotFileBatchSource	all properties*	Creates dataset from properties.	Dataset will be incorrect with macro literals as fields.
StreamBatchSource	schema, name, format	Stream is added and created through name and schema is parsed to set output schema.	Macro literals will lead to bad parsing of properties.
TableSource	schema	Schema parsed to set output schema.	Macro literals will lead to failed or incorrect schema creation.
TimePartitionedFileSetDatasetAvroSource	schema	Schema parsed to set output schema.	Macro literals will lead to failed or incorrect schema creation.
TimePartitionedFileSetDatasetParquetSource	schema	Schema parsed to set output schema.	Macro literals will lead to failed or incorrect schema creation.
TimePartitionedFileSetSource	name, basePath	Name and basePath are used to create the dataset.	Macro literals can lead to wrong or bad dataset creation.
JavaScriptTransform	JavaScriptTransform	schema, script, lookup	Schema format is used to set the output schema. JavaScript and lookup properties are also parsed for correctness.	Macro literals can cause parsing to fail for schema creation, JavaScript compilation, or lookup parsing.
LogParserTransform	inputName	Gets field from input schema through inputName property.	With a macro literal, the field will not exist in the input schema.
ProjectionTransform	fieldsToKeep, fieldsToDrop, fieldsToConvert, fieldsToRename	Properties are used to create output schema.	Macro literals will lied to a failed or wrong output schema being created.
PythonEvaluator	schema	Schema parsed for correctness and set as output schema.	Macro literal can lead to failed or bad schema creation.
ValidatorTransform	validators, validationScript,	Validator property used to set validator plugins. Script property is also parsed for correctness.	Macro literals can lead to failed parsing or plugins being set. Scripts can not be validated without validators.
ElasticsearchSource	schema	Schema parsed for correctness and set as output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
HBaseSink	rowField, schema	Parsed to valid the output and input schemas and set the ouput schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
HBaseSource	schema	Parsed for correctness to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
HiveBatchSource	schema	Parsed for correctness to set ouput schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
MongoDBBatchSource	schema	Parsed for correctness and validated to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
NaiveBayesClassifier	predictionField	Configures and sets fields of output schema and checked for existence in input schema.	Output schema would be created wrongly with macro literal as prediction field and input schema check behavior is undefined.
Compressor	compressor, schema	Parsed for correctness and used to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
CSVFormatter	schema	Parsed for correctness and used to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
CSVParser	field	Validated against input schema to check existence of field.	Macro literals may not exist as fields in the input schema.
Decoder	decode, schema	Decode property is parsed and validated then used to validate the input schema. Schema parsed to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation or incorrect validation of input schema.
Decompressor	decompressor, schema	Decompressor property is parsed and validated then used to validate the input schema. Schema parsed to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation or incorrect validation of input schema.
Encoder	encode, schema	Encode property is parsed and validated then used to validate the input schema. Schema parsed to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation or incorrect validation of input schema.
JSONFormatter	schema	Parsed for correctness and used to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.
JSONParser	field, schema	Validates if field property is present in input schema. Parses schema property to set output schema.	Macro literal may not exist in input schema and may lead to failed parsing or creation of output schema.
StreamFormatter	schema	Parsed for correctness and used to set output schema.	Macro literals can lead to failed or incorrect schema parsing/creation.

...

Other plugins have fields that are validated/processed at configure time that do not affect the schema. In these cases, these can be moved to initialization timethe prepare run method. The following plugins and fields would be affected:

Plugin	Fields	Use	Justification
StreamBatchSource	duration, delay	Parsed and validated for proper formatting.	The parsing/validation is not related to the schema's creation.
TimePartitionedFileSetSource	duration, delay	Parsed and validated for proper formatting.	The parsing/validation is not related to the schema's or dataset's creation.
FileBatchSourceReferenceBatchSink	timeTable	Creates dataset from time table property.	The dataset creation does not change the schema.	BatchWritableSink	name, type	Creates dataset from properties.	The dataset creation does not change the schemareferenceName	Verifies reference name meets dataset ID constraints.	As dataset names can be macros, this supports the primary use case.
ReferenceBatchSource	referenceName	Verifies that reference name meets dataset ID constraints.	As dataset names can be macros, this supports the primary use case.
FileBatchSource	timeTable	Creates dataset from time table property.	This is a primary use case for macros.
TimePartitionedFileSetSource	name, basePath	Name and basePath are used to create the dataset.	This is a primary use case for macros.
BatchWritableSink	name, type	Creates dataset from properties.	This is a primary use case for macros.
SnapshotFileBatchSink	name	Creates dataset from name field.	This is a primary use case for macros.
BatchReadableSource	name, type	Dataset is created from name and type properties.	This is a primary use case for macros.
SnapshotFileBatchSource	all properties*	Creates dataset from properties.	This is a primary use case for macros.
TimePartitionedFileSetSink	all properties*	Creates dataset from properties.	This is a primary use case for macros.
DBSource	importQuery, boundingQuery, splitBy, numSplits	Validate connection settings and parsed for formatting.	The parsing/validation does not lead to the creation of any schema or dataset.
HDFSSink	timeSuffix	Parsed to validate proper formatting of time suffix.	The parsing/validation does not lead to the creation of any schema or dataset.
KafkaProducer	async	Parsed to check proper formatting of boolean.	The parsing/validation does not lead to the creation of any schema or dataset.
NaiveBayesClassifier	fieldToClassify	Checked if input schema field is of type String.	The validation does not lead to the creation or alteration of any schema.
NaiveBayesTrainer	fieldToClassify, predictionField	Checked if input schema fields are of type String and Double respectively.	The validation does not lead to the creation or alteration of any schema.
CloneRecord	copies	Validated against being 0 or over the max number of copies.	The validation does not lead to the creation of any schema or dataset.
CSVFormatter	format	Validated for proper formatting.	The validation does not lead to the creation of any schema or dataset.
CSVParser	format	Validated for proper formatting.	The validation does not lead to the creation of any schema or dataset.
Hasher	hash	Checked against valid hash formats.	The check does not lead to the validation or alteration of any schema.
JSONParser	mapping	Mappings extracted and placed into a map with their expressions.	The extraction does not affect any schema creation or validation.
StreamFormatter	format	Checked against valid stream formats.	The check does not lead to the validation or alteration of any schema.
ValueMapper	mapping, defaults	Parsed after configuration is initialized and validated.	The check does not lead to the validation or alteration of any schema.

...