Overview 

This addition will allow users to see the history of directives made to a column of data.

Goals

User should be able to see lineage information, ie. directives, for columns

Storing lineage information should have minimal/no impact to the wrangler application

User Stories

• As a user, I should be able to see the directives applied to a column of data.

• As a user, I should be able to see the directives applied to a column of data over any period of time.

• As a user, I should be able to add tags and properties to specific columns of data (stretch)

Design

Save directives for each column in AST format during execution of wrangler along with necessary information (time, dataset/stream name/id).

Use TMS to send information to platform.

Unmarshal and store in HBase.

Access to lineage should only be available through the platform

Questions

• How to get source and sink datasets?
• How to handle set columns step?

Approach

Approach #1:

Store directives during execution of each step

Advantages:

• No switch statement; DirectivesLineage class will not need to be modified with new directives
• Would integrate well with user defined directive and other applications in future

Disadvantages:

• Add getter to each step class + sometimes (~30%) local variable
• Slower

Approach #2 (Preferred):

Compute lineage without looking at data by backtracking

Advantages:

• Wouldn't need to modify any step functions
• Faster

Disadvantages:

• Will require large Switch or If/Else statement
• Breaks with some directives (See * below); requires stricter rule on directives, ie. every rename must give old and new name
• More assumption based, ie. parse-as-<> assumes that the output fields are from all the input fields
• Any new directive would have to change DirectivesLineage class
• Wouldn't integrate well with user defined directives

*Backtrack starting with columns A,B,C. Previous directive is "set-columns A B C". The directive before that is "filter-row-if-matched <column> <regex>" where <column> is nameOfOwner. No way of knowing what nameOfOwner refers to without looking at data.

API changes

New Programmatic APIs

DirectivesLineage Java class that contains all necessary information to be sent to CDAP platform

Instance should be initialized at Wrangler step; should be parsed along with execute function.

 public class DirectivesLineage {
  private final long startTime;
  private final String programName;
  private int numberOfColumns;
  private String dataSetName; //Dataset or Stream name/id
  private Map<String, ColumnDirectives> lineage;

  public DirectivesLineage(String dataSetName, String[] columnNames) {
    this.startTime = System.currentTimeMillis();
    this.programName = "wrangler";
    this.dataSetName = dataSetName;
    this.numberOfColumns = columnNames.length;
    this.lineage = new HashMap<>(columnNames.length);
    for (int i = 0; i < columnNames.length; ++i)
      lineage.put(columnNames[i], new ColumnDirectives(columnNames[i], i));
    this.branches = new HashMap<>();
  }

  // getters for startTime, programName, dataSetName
  // setter for dataSetName

  // helper functions for parse

  public void store(Step currStep) {
	/*
    switch(type of currStep) {
      Handle each case and store in lineage accordingly
    }
	*/
  }

  private class ColumnDirectives {
    final String originalName; // If null it is a new column, for linking in database
    int colNum; // negative if not in workspace;
    int version = NEW_DIR; // for making copies
    List<MetaStep> steps = new ArrayList<>();  // AST that stores directives for this column

    // constructors
  }

  private class MetaStep {
    Step directive;
    List<String> colNames; // name of column
    List<String> cdNames; // name of column in branches

    // constructors
  }
}

New REST APIs

/v3/namespaces/{namespace-id}/datasets/{dataset-id}/columns/{column-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels>

/v3/namespaces/{namespace-id}/streams/{stream-id}/columns/{column-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels>

CLI Impact or Changes

TBD

UI Impact or Changes

• Add interface to metadata table when viewing dataset to see lineage of columns possibly by clicking on column:
  
• When a column is clicked on will look something like:
  

Security Impact 

Should be none, TBD

Impact on Infrastructure Outages 

Storage in HBase; Impact TBD.

Test Scenarios

Releases

Release 4.3.0

Release 4.4.0

Related Work

• Fixing TextDirectives and parsing of directives in general