Page Comparison

...

/**
 * Contains information about a property used by a plugin.
 */
@Beta
public class PluginPropertyField {

  private final String name;
  private final String description;
  private final String type;
  private final boolean required;
  // returns true if this field can accept macro
  private final boolean macroEnabled;
  ...
}

Notes

This will require a CDAP platform level change as it's a new annotation.

...

During configure time in the configurePipeline method, if a field is macro enabled, the property should not be validated as the macro has not been provided a substitutable value.

Good Place to put isResolved method

We don't want to force deferring macros to runtime in the case that a field is macroable but has no macro substituted.To allow macro substitution of non-String properties, any properties configured with a macro will have a placeholder default value at configure time. For primitive types, this would be Java's default value. For objects, this would be null. This is ok because we are exposing a method to check whether a property is safe to validate at configure time through the pipelineConfigurer object.

The methods to check property safety is also helpful in allowing plugin developers to determine whether or not a dataset should be created at runtime (whether it was already created at configure time or not).

PluginConfigurer Changes:

PluginConfigurer will expose the following field

public interface PluginConfigurer extends DatasetConfigurer {
  // ...
 
  boolean isMacro(String fieldName);
}

The method will return whether or not the property with the provided fieldName contained a macro at configure time. We don't want to force deferring macros to runtime in the case that a field is macroable but actually has no macro provided in its configuration. This allows optional checking of properties at configure time:

@Override
void configurePipeline(PipelineConfigurer pipelineConfigurer) {
  if (!pipelineConfigurer.isMacro("name")) {
    // perform some operation using the "name" property
  }
  // ...
}

 To allow macro substitution of non-String properties, any properties configured with a macro will have a placeholder default value at configure time. For primitive types, this would be Java's default value. For objects, this would be null. This is ok because we are exposing a method to check whether a property is safe to validate at configure time through the pipelineConfigurer object.

Given that a dataset could be created at configure time if no fields are provided macros, a check should be available for plugin developers to see whether the dataset already exists at runtime. We can do this by altering the runtime context object passed in to the prepareRun method. As the object extends BatchContext which extends DatasetContext, we can create a new method in BatchContext that checks for the existence of a dataset.

public interface BatchContext extends DatasetContext, TransformContext {
	// ...
 
	boolean datasetExists(String datasetName);
}

This method will return whether or not the dataset with the provided datasetName already exists and can be used in prepareRun:

@Override
public void prepareRun(BatchSinkContext context) {
  if (!context.datasetExists(config.getName())) {
    pipelineConfigurer.createDataset(config.getName(), ...);
  }
  // ...
}

During runtime, PipelineInsantiator would get config fields and values to substitute and can use that information to substitute macro appropriately and return an instantiated plugin.

"plugin": {
	"name": "Database",
	"type": "batchsource",
	"properties": {
		"user": "${wf-token(username)}",
		"password": "${secure(sql-password)}",
		"jdbcPluginName": "jdbc",         
		"importQuery": "select * from ${wf-token(table-name)};"
	}
}

interface MacroContext {	
	/**
	 * Given the macro key, return the substituted value
     */ 
	String getValue(String macroKey);
}

Based on the macro type, one of the below MacroContext's will be used to get the value for macro. 
 
DefaultMacroContext implements MacroContext {
	Map<String, String> runtimeArguments;
	String getValue(String macroKey) {
		return runtimeArguments.get(macroKey);
	}
}

SecureMacroContext implements MacroContext {
	SecureStore secureStore;
	String getValue(String macroKey) {
		return secureStore.get(macroKey);
	}
}

RuntimeFunctionMacro implements MacroContext {	
	long logicalStartTime;
	Function<String, String> timezoneFunction;
	String getValue(String arguments) {
		return timezoneFunction.apply(arguments);
	}
} 

...