Introduction Introduction

This design provides the capability for passing information between the triggering pipeline and triggered pipeline based on the program status based scheduling.

Goals

Provides clear API and backend design for passing program status change event payload between pipelines. 

User Stories 

• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the username and password used by Pipeline A
• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the stream id Pipeline A reads from
• Pipeline B is triggered when Pipeline A completes, and Pipeline B also needs the schema of a sink Pipeline A writes to.
  
  

Design

When setting up a program status schedule, users need to specify which runtime args from the triggering pipeline will be used in the triggered Pipeline, whether to include all stage configurations from the triggering pipeline, whether to include all user tokens from the triggering pipeline. All the needed runtime argsA new method will be provided in RuntimeContext to return a new API class TriggeringScheduleInfo to provide information of the schedule which launches the program. All the runtime arguments, stage configurations, user workflow tokens from of the triggering pipeline will be attached with a certain prefix according to their type and pipeline info, and will be passed to the triggered pipeline as runtime args. Optionally, a map that specifies which runtime args from the triggering pipeline will be used as which runtime args in the current pipeline can also be configured.run which triggers the schedule will be included in the . To use these properties, users can define a mapping between a certain property from the triggering program and a runtime argument in the triggered program. Such mapping should be stored in the schedule properties with the key "triggering.properties.mapping". The app containing the triggered program should be able to decode the value contained in the field "triggering.properties.mapping" and get the corresponding properties from TriggeringScheduleInfo and use the values as runtime arguments according to the properties mapping.

For pipeline app, the syntax for referring to different properties in the  filed "triggering.properties.mapping" is defined below:

The syntax for runtime args from the triggering pipeline is: runtime-arg:<namespace>:<pipeline-version>:<pipeline-name>#<runtime-arg-key>

The syntax for stage Stage configuration from the triggering pipeline is: stagepipeline-config:<namespace>:<pipeline-version>:<pipeline-name>#<pipeline-stage>:<stage-config-key>

The syntax for user User tokens from the triggering pipeline is: token:<namespace>:<pipeline-version>:<pipeline-name>#<node-name>:<user-token-key>When user develop When the schedule launches the pipeline, they can access runtime args, stage configurations, user tokens from the triggering pipeline directly from by constructing the runtime arg key with the correct syntax. Or they can map the payload from triggering pipeline to an existing runtime arg in the triggered pipeline. 

Approach

Here's an example use case to illustrate the approach: Pipeline A in namespace Default triggers Pipeline B when Pipeline A completes. Pipeline B needs the runtime args hostname from Pipeline A as the value of its host field in runtime args. For instance, For instance, Pipeline A contains runtime args pair "hostname" -> "wiki.cask.co" , then in Pipeline B, the value of "host" should be "wiki.cask.co"

When setting up program status schedule, a map an entry "runtime-arg:default:A#hostname" -> "host" will be stored in the schedule. When Pipeline A completes, it adds a key-value pair "runtime-arg:default:A:-SNAPSHOT#host" : "wiki.cask.co"  to the notification. When this notification triggers the schedule to launch Pipeline B, the launcher look for the value of filed a map, and this map will be converted to a JSON String to be stored as a value of the field "triggering.properties.mapping" in the schedule properties. When the notification of Pipeline A's completion triggers the schedule to launch Pipeline B, in the Pipeline app of Pipeline B, the field "triggering.properties.mapping" is extracted from schedule properties. From the decoded map, Pipeline app recognizes the key "runtime-arg:default:A:-SNAPSHOT#host" in the notification and use the value in runtime args pair "hostname" -> "wiki.cask.co" and pass it to run Pipeline B.#hostname" means a runtime argument with key "hostname" in the Pipeline with name A  in the default namespace. Pipeline app thus looks this runtime argument value in Pipeline A from TriggeringScheduleInfo from the RuntimeContext , and provides it as the value for runtime argument "host" in Pipeline B

API changes

New Programmatic APIs

Instead of letting user directly specifying the mapping, provides a builder to build it for users. 

/**
 * CreateThis ainterface schedulerepresents whicha iscontext triggeredfor whena theprocessor givenor programelements inof the given namespace, application, anda processor.
 */
applicationpublic versioninterface transitionsRuntimeContext to{
any
one...
of
the given program statuses.
 *
 * @param programNamespace the namespace where this program is defined
 * @param application the name of the application where this  /**
   * @return {@link TriggeringScheduleInfo} if the program is definedtriggered by *a @paramschedule. appVersionOtherwise, thereturns version of the application{@code null}.
   */
@param programType the@Nullable
type of the program, as supported by the systemTriggeringScheduleInfo getTriggeringScheduleInfo();
 
...
 
}

/**
 * @param program the nameThe information of thea programschedule to *be @parampassed programStatusesto the setprogram oflaunched statusesby toit
trigger the*/
schedule.public Theclass scheduleTriggeringScheduleInfo will{
be
triggered if theprivate statusfinal ofString name;
*  private final String description;
  private final Map<String, String> properties;
  private final Map<String, List<ApplicationSpecification>> namespaceApplicationSpecificationsMap;
  private final theMap<String, specificWorkflowToken> programuserWorkflowTokenMap;
transitioned to oneprivate offinal these statuses.
 * @return a {@link ScheduleCreationSpec}
 */
RuntimeArgsBuilder triggerOnProgramStatus(String programNamespaceMap<String, Map<String, String>> runtimeArgumentsMap;

  public TriggeringScheduleInfo(String name, String applicationdescription, String appVersionMap<String, String> properties, 
                                       ProgramType programTypeMap<String, StringList<ApplicationSpecification>> programnamespaceApplicationSpecificationsMap, 
                                Map<String, WorkflowToken> userWorkflowTokenMap, 
     ProgramStatus... programStatuses);

/**  * A builder to build a schedule triggered by.  */ public interface ProgramStatusTriggerBuilder implements TriggerBuilder {   /**    * AddsMap<String, aMap<String, runtimeString>> argsruntimeArgumentsMap) mapping{
between a runtime arg fromthis.name the= triggeringname;
program    *this.description to= bedescription;
used as the value ofthis.properties a= runtimeproperties;
arg in the current programthis.namespaceApplicationSpecificationsMap = namespaceApplicationSpecificationsMap;
 *   this.userWorkflowTokenMap = *userWorkflowTokenMap;
@param keyFromTriggeringProgram the name ofthis.runtimeArgumentsMap the= runtimeruntimeArgumentsMap;
arg from the}
triggering
pipeline whose value/**
will be used * @return Schedule's *name, which is unique in an application.
   */
  public String getName() {
    return name;
  }

  /**
   as* the@return valueDescription of the schedule.
given  runtime arg*/
in the currentpublic pipelineString getDescription() {
 * @param key thereturn namedescription;
runtime arg in}
the
current pipeline /**
   * @return aProperties {@linkof ProgramStatusTriggerBuilder} containing all existing configurationsthe schedule.
   */
  ProgramStatusTriggerBuilder withRuntimeArg(String keyFromTriggeringProgrampublic Map<String, StringString> keygetProperties();  {
  /**  return properties;
* Adds a}
runtime
args mapping between/**
a runtime arg from* the@return triggeringA programmap with namespace names *as tokeys, beand used as the valuelist of aapplication runtimespecification arg inof the currentparent program.applications
   *     * @param Map<String, String> runtimeArgsMap thewhich map containingcontain the runtimeprograms argthat fromtrigger the triggering pipelineschedule as keysvalues.
   */
  Map<String, List<ApplicationSpecification>> getNamespaceApplicationSpecificationsMap() {
    return namespaceApplicationSpecificationsMap;
  }

  /**
   * @return A map with workflow ID String as keys in the format of 
   *  and runtime arg in the current pipeline as values
   * @return a {@link ProgramStatusTriggerBuilder} containing all existing configurations
   */
  ProgramStatusTriggerBuilder withRuntimeArgsMap(Map<String, String> runtimeArgsMap);
 
  /**
   * Passes all the stage configurations from the triggering pipeline as runtime arguments to the triggered pipeline
   * as runtime args "namespace:application:application-version:workflow-name", or  "namespace:application:workflow-name" 
   *         if the application version is of default version "-SNAPSHOT", and the corresponding user-scope 
   *         workflow token as values.
   */
  public *Map<String, @returnWorkflowToken> agetUserWorkflowTokenMap() {@link
ProgramStatusTriggerBuilder} containing all existing configurationsreturn  userWorkflowTokenMap;
 */   ProgramStatusTriggerBuilder useTriggeringStageConfigs();}

  /**
   * Passes@return allA themap userwith tokensprogram fromID theString triggeringas pipelinekeys asin runtimethe argumentsformat toof the
triggered pipeline  *  * as runtime args.    *"namespace:application:application-version:program-type:program-name", 
   *   @return a {@link ProgramStatusTriggerBuilder} containing all existingor configurations
   */
  ProgramStatusTriggerBuilder useTriggeringUserTokens();
 
  /**
   * Builds the schedule containing the program status trigger with "namespace:application:program-type:program-name" if the application version is of 
   *         default version "-SNAPSHOT", and the runtime argsarguments mappingkey-value pairs as  * values.
   */
@return a {@link ProgramStatusTrigger} containing the program status trigger with the runtime args mapping
   */
  ProgramStatusTrigger build();public Map<String, Map<String, String>> getRuntimeArgumentsMap() {
    return runtimeArgumentsMap;
  }
}

UI Impact or Changes

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work