SparkTransform

User should be able to run a spark job that operates as a transform in ETL Batch scenario.


API Design:

Below is the SparkTransform class that users can implement to define their Spark transform plugin.
Because it extends PipelineConfigurable, users will be able to override:

• configurePipeline - in order to add datasets and streams needed

The SparkMLLib also exposes a transform method, which gives access to an RDD representing data from previous stages of the pipeline.

• transform - perform any transformations and return an RDD representing the transformed RDD.
  
  

/**
 * Spark Transform stage.
 *
 * @param <IN> Type of input object
 * @param <OUT> Type of output object
 */
@Beta
public abstract class SparkTransform<IN, OUT> implements PipelineConfigurable, Serializable {
  public static final String PLUGIN_TYPE = "sparktransform";

  private static final long serialVersionUID = -8156450728774382658L;


  /**
   * Configure an ETL pipeline.
   *
   * @param pipelineConfigurer the configurer used to add required datasets and streams
   * @throws IllegalArgumentException if the given config is invalid
   */
  @Override
  public void configurePipeline(PipelineConfigurer pipelineConfigurer) throws IllegalArgumentException {
    //no-op
  }

  /**
   * Transform the input and return the output to be sent to the next stage in the pipeline.
   *
   * @param context {@link SparkPluginContext} for this job
   * @param input input data to be transformed
   * @throws Exception if there's an error during this method invocation
   */
  public abstract JavaRDD<OUT> transform(SparkPluginContext context, JavaRDD<IN> input) throws Exception;

}

Users will have access to a

SparkPluginContext

 which exposes functionality that would be available to a regular Spark program via SparkContext, except it excludes the following methods.:

• getMetrics
• getWorkflowToken
• getTaskLocalizationContext
• getSpecification
• getServiceDiscoverer
• setExecutorResources

Implementation Summary:

1. SmartWorkflow will break up any stages of SparkTransform type into its own phase to be run in ETLSpark, instead of ETLMapReduce. It will be broken up into its own phase because ETLSpark doesn't support certain sources and transforms.
2. There will be special casing in ETLSparkProgram to call the user's SparkTransform class's transform method after reading from the source and before writing to the sink.