What is Cask Tracker ?

Cask Tracker (formerly Cask Finder) is a CDAP Extension that provides the ability to track data ingested either through Cask Hydrator or a Custom CDAP Application and provide input to data governance processes on a cluster. 

It includes this data about "the data":

• Metadata
  • Tags, Properties, Schema for CDAP Datasets and Programs
  • System and User 
• Data Quality
  • Metadata that include Feed-level and Field-level quality metrics of datasets
• Data Usage Statistics 
  
  • Usage statistics of dataset and programs. 

Checklist

• User stories documented (Nitin)
• User stories reviewed (Nitin)
• Design documented (Shankar)
• Design reviewed (Terence/Andreas/Albert)
• Feature merged (Shankar)
• UI updated (Ajai/Edwin)
• Documentation for feature (Shankar)

Use-cases

Following are the use-cases for the Cask Tracker.

Case #1

• Rishab is a data scientist/engineer at a company that implements a Data Lake. He is analyzing the effectiveness of the recommendation engine on the company's e-commerce site. For this investigation, he wants to analyze a dataset that includes click log for the last year. He is looking for clean click log data that is up-to-date. He wants to use part of the data to build model and rest to score the model and validate the predictions.  
• Before he can conduct an analysis, Rishab needs to confirm the dataset is available in the Data Lake.
• To do so, he wishes to find all entities that include “click log”.
• He arrives at the Finder home screen (from nav, search results, other entry points?).
  • Enters “click log” in the Search Box and clicks Search.
  • He arrives at the Results Page. 
    • Results returned
    • By default, they are sorted by creation time
    • Each Result includes:
      • Snippet of the metadata that matches his query in context.
        • Important to help him evaluate the relevance of the results.
      • Date Created
        • To know how recent/new it is.
• For this analysis, Rishab is most concerned with the recency, the accuracy, and the integrity of the data.
• He clicks the result and arrives at the Entity Detail Page where he can view all of the metadata associated with an entity. 
• Rishab wished to verify the validity of the sources of this dataset. To do so, he clicks the Lineage Tab to trace the creation of this dataset to its source.
• Finder displays the lineage for this dataset as a diagram. The selected dataset displays in the center; to the left is the entity that precedes it and to the right is the one it precedes.
• Rishab discovers that it has been created from two separate sources.
• He then clicks one of the sources which takes him to the Entity Page of that dataset.
• He clicks on a program to see what has been done to the dataset.
• Rishab clicks the Audit Logs Tab to see how active this dataset has been - when was it last updated, who is using it, writing to it, reading from it.
• Rishab clicks the appropriate action to make this dataset a new source for his existing Click Log processing pipeline.
• This takes him to the Hydrator Studio where he can edit the Master Click Log pipeline.

Case #2 (Incomplete)

• D-Rock, an IT genius, has been tasked with finding out all the datasets on the cluster that have the field SSN.
• He needs to find every dataset with an SSN field and mask the field by generating new feed containing a masked SSN. 
• In order to achieve this task, he uses Cask Finder to achieve this task
  • Enters “SSN” in the Search Box and clicks Search.
  • He then filters by Schema – meaning he would only like to see results which includes datasets where the SSN field is matched in Schema.
• He then selects the dataset that has this field and ...

Design

Release 1.0 Deliverables

Tracker API Meeting Notes: 2016-03-01

Initial design for audit log API

Endpoint:

• GET /auditlog/<dataset>?starttime, endtime

MUST support:

• relative dates (1d, 1h, 1m, alltime)
• pagination

MUST return:

• total results
• timestamp
• user
• action type
• kind of entity
• name of entity
• entity metadata

Example:

MapReduce job "X" accesses dataset "A" for reading:

timestamp: t0
user: default
action type: READ
kind of entity: MAPREDUCE
name of entity: X
entity meta: { runid: ... }