Introduction 

This design provides the capability for passing information between the triggering pipeline and triggered pipeline based on the program status based scheduling.

Goals

Provides clear API and backend design for passing program status change event payload between pipelines. 

User Stories 

• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the username and password used by Pipeline A
• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the stream id Pipeline A reads from
• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the schema of a sink Pipeline A writes to.
  
  

Design

When setting up a program status schedule, users need to specify which runtime args from the triggering pipeline will be used in the triggered Pipeline, whether to include all stage configurations from the triggering pipeline, whether to include all user tokens from the triggering pipeline. All the needed runtime args, stage configurations, user tokens from the triggering pipeline will be attached with a certain prefix according to their type and pipeline info, and will be passed to the triggered pipeline as runtime args. Optionally, a map that specifies which runtime args from the triggering pipeline will be used as which runtime args in the current pipeline can also be configured.

The syntax for runtime args from the triggering pipeline is: runtime-arg:<namespace>:<pipeline-version>:<pipeline-name>#<runtime-arg-key>

The syntax for stage configuration from the triggering pipeline is: stage-config:<namespace>:<pipeline-version>:<pipeline-name>#<pipeline-stage>:<stage-config-key>

The syntax for user tokens from the triggering pipeline is: token:<namespace>:<pipeline-version>:<pipeline-name>#<node-name>:<user-token-key>

When user develop When the schedule launches the pipeline, they can access runtime args, stage configurations, user tokens from the triggering pipeline directly from by constructing the runtime arg key with the correct syntax. Or they can map the payload from triggering pipeline to an existing runtime arg in the triggered pipeline.

Approach

Here's an example use case to illustrate the approach: Pipeline A in namespace Default triggers Pipeline B when Pipeline A completes. Pipeline B needs the runtime args hostname from Pipeline A as the value of its host field in runtime args. 

When setting up program status schedule, a map entry "hostname" -> "host" will be stored in the schedule. When Pipeline A completes, it adds a key-value pair "runtime-arg:default:A:-SNAPSHOT#host" : "wiki.cask.co"  to the notification. When this notification triggers the schedule to launch Pipeline B, the launcher look for the value of filed "runtime-arg:default:A:-SNAPSHOT#host" in the notification and use the value in runtime args pair "hostname" -> "wiki.cask.co" and pass it to run Pipeline B.

API changes

New Programmatic APIs

Instead of letting user directly specifying the mapping, provides a builder to build it for users.

/**
 * Create a schedule which is triggered when the given program in the given namespace, application, and
 * application version transitions to any one of the given program statuses.
 *
 * @param programNamespace the namespace where this program is defined
 * @param application the name of the application where this program is defined
 * @param appVersion the version of the application
 * @param programType the type of the program, as supported by the system
 * @param program the name of the program
 * @param programStatuses the set of statuses to trigger the schedule. The schedule will be triggered if the status of
 *                        the specific program transitioned to one of these statuses.
 * @return a {@link ScheduleCreationSpec}
 */
RuntimeArgsBuilder triggerOnProgramStatus(String programNamespace, String application, String appVersion,
                                          ProgramType programType, String program,
                                          ProgramStatus... programStatuses);

/**
 * A builder to build a schedule triggered by.
 */
public interface ProgramStatusTriggerBuilder implements TriggerBuilder {
  /**
   * Adds a runtime args mapping between a runtime arg from the triggering program
   * to be used as the value of a runtime arg in the current program.
   * 
   * @param keyFromTriggeringProgram the name of the runtime arg from the triggering pipeline whose value will be used
   *                                 as the value of the given runtime arg in the current pipeline
   * @param key the name runtime arg in the current pipeline 
   * @return a {@link ProgramStatusTriggerBuilder} containing all existing configurations
   */
  ProgramStatusTriggerBuilder withRuntimeArg(String keyFromTriggeringProgram, String key);
 
  /**
   * Adds a runtime args mapping between a runtime arg from the triggering program
   * to be used as the value of a runtime arg in the current program.
   * 
   * @param Map<String, String> runtimeArgsMap the map containing the runtime arg from the triggering pipeline as keys
   *                                           and runtime arg in the current pipeline as values
   * @return a {@link ProgramStatusTriggerBuilder} containing all existing configurations
   */
  ProgramStatusTriggerBuilder withRuntimeArgsMap(Map<String, String> runtimeArgsMap);
 
  /**
   * Passes all the stage configurations from the triggering pipeline as runtime arguments to the triggered pipeline
   * as runtime args.
   *
   * @return a {@link ProgramStatusTriggerBuilder} containing all existing configurations
   */
  ProgramStatusTriggerBuilder useTriggeringStageConfigs();
  /**
   * Passes all the user tokens from the triggering pipeline as runtime arguments to the triggered pipeline
   * as runtime args.
   * 
   * @return a {@link ProgramStatusTriggerBuilder} containing all existing configurations
   */
  ProgramStatusTriggerBuilder useTriggeringUserTokens();
 
  /**
   * Builds the schedule containing the program status trigger with the runtime args mapping
   * 
   * @return a {@link ProgramStatusTrigger} containing the program status trigger with the runtime args mapping
   */
  ProgramStatusTrigger build();
}

UI Impact or Changes

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work