Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Current »

This is a sub-task (single user story) of Cask Hydrator++.

JIRA: 

Error rendering macro 'jira' : Unable to locate Jira server for this macro. It may be due to Application Link configuration.

 

User Story:

User should be able to run a spark job that operates as a transform in ETL Batch scenario.


API Design:

Below is the SparkTransform class that users can implement to define their Spark transform plugin.
Because it extends PipelineConfigurable, users will be able to override:

  • configurePipeline - in order to add datasets and streams needed

The SparkMLLib also exposes a transform method, which gives access to an RDD representing data from previous stages of the pipeline.

  • transform - perform any transformations and return an RDD representing the transformed RDD.

SparkTransform
/**
 * Spark Transform stage.
 *
 * @param <IN> Type of input object
 * @param <OUT> Type of output object
 */
@Beta
public abstract class SparkTransform<IN, OUT> implements PipelineConfigurable, Serializable {
  public static final String PLUGIN_TYPE = "sparktransform";

  private static final long serialVersionUID = -8156450728774382658L;


  /**
   * Configure an ETL pipeline.
   *
   * @param pipelineConfigurer the configurer used to add required datasets and streams
   * @throws IllegalArgumentException if the given config is invalid
   */
  @Override
  public void configurePipeline(PipelineConfigurer pipelineConfigurer) throws IllegalArgumentException {
    //no-op
  }

  /**
   * Transform the input and return the output to be sent to the next stage in the pipeline.
   *
   * @param context {@link SparkPluginContext} for this job
   * @param input input data to be transformed
   * @throws Exception if there's an error during this method invocation
   */
  public abstract JavaRDD<OUT> transform(SparkPluginContext context, JavaRDD<IN> input) throws Exception;

}


Users will have access to a

SparkPluginContext

 which exposes functionality that would be available to a regular Spark program via SparkContext, except it excludes the following methods.:

  • getMetrics
  • getWorkflowToken
  • getTaskLocalizationContext
  • getSpecification
  • getServiceDiscoverer
  • setExecutorResources


Implementation Summary:

  1. SmartWorkflow will break up any stages of SparkTransform type into its own phase to be run in ETLSpark, instead of ETLMapReduce. It will be broken up into its own phase because ETLSpark doesn't support certain sources and transforms.
  2. There will be special casing in ETLSparkProgram to call the user's SparkTransform class's transform method after reading from the source and before writing to the sink.
  • No labels