Page Comparison

...

This also includes removing the temporary datasets that could be created as a part of workflows.

API Changes: 

Approach Choices for APIs: 

 #1 Programtype rollup=workflow: For this API, programs New argument added for the lineage API. Programs in all relations are simply replaced by their associated workflows, if applicableany. This means that this lineage Programs that are not part of any workflows are left as it is. 

Lineage view will not have a mapping of between programs to and workflows. 

• JSON Response Changes: 

1. In the "programs" section: the ProgramId of the Program will be replaced by ProgramId of the workflow if the program is associated with a workflow. If not the Programs show as it is. 
2. In the "relations" section: the programs field will carry the name of the workflows if applicable.

[Example shown below]

...

• JSON Response Changes: 

...

#3 [ NOT considering now ] Group-by on Workflow: In addition to the collapse, if a group-by is provided, it can be used as group-by=workflow and the programs that dont have workflows associated are left as it is. This approach also has some level of ambiguity listed above but is extensible if in future there is a requirement for group-by on application. [Because collapse workflow is confusing if there are no workflows present]  

Implementation Changes: 

1. ....

Implementation Changes: 

1. Once all the relations are set up for the required lineage, the new code: 
   1. Walks over all relations and makes a list of all Workflow IDs associated with all Programs in the relations. 
   2. For all these workflow IDs, AppMetadataStore is scanned to return a map of <ProgramID, MDSkey>. [while applying the workflow IDs as filter]
   3. Walks this map and creates another map of <RunIDs, ProgramIDs>. This map contains all workflow RunIDs for associated programs in the required relations. 
   4. Walk over the relations again, and replace ProgramIDs of Programs with ProgramIDs of associated workflows.  
   
   

curl "http://127.0.0.1:10000/v3/namespaces/default/datasets/EmpAgg/lineage?collapse=access&collapse=run&collapse=component&programtype=workflow&end=now&levels=1&start=now-7d" | python -m json.tool

...

curl "http://127.0.0.1:10000/v3/namespaces/default/datasets/EmpAgg/lineage?collapse=access&collapse=run&collapse=component&programtype=workflow&end=now&levels=1&start=now-7d" | python -m json.tool


{
    "data": {
        "dataset.default.EmpAgg": {
            "entityId": {
                "id": {
                    "instanceId": "EmpAgg",
                    "namespace": {
                        "id": "default"
                    }
                },
                "type": "datasetinstance"
            }
        },
        "dataset.default.conn-0.e0591f36-9661-11e6-af4a-0000007182af": {
            "entityId": {
                "id": {
                    "instanceId": "conn-0.e0591f36-9661-11e6-af4a-0000007182af",
                    "namespace": {
                        "id": "default"
                    }
                },
                "type": "datasetinstance"
            }
        }
    },
    "end": 1476926333,
    "programs": { 
        "<workflow name>": {
            "entityId": {
                "id": {
                    "application": {
                        "applicationId": "EmployeePipe_Long_copy",
                        "namespace": {
                            "id": "default"
                        }
                    },
                    "id": "phase-2",
                    "type": "Mapreduce"
                },
                "type": "<type workflow>"
            }
        }
    },
    "relations": [
        {
            "accesses": [
                "read", 
                "write"
            ],
            "components": [],
            "data": "dataset.default.conn-0.e0591f36-9661-11e6-af4a-0000007182af",
            "program" : "<workflow name>"
            "runs": [
                "e4051038-9661-11e6-8060-000000d79ea8",
                "e4051038-9661-11e6-8060-000000d79ea8"
            ]
        },
        {
            "accesses": [
                "write",
                "read"
            ],
            "components": [],
            "data": "dataset.default.EmpAgg",
            "program": "mapreduce.default.EmployeePipe_Long_copy.phase-2",
            "runs": [
                "e4051038-9661-11e6-8060-000000d79ea8"
            ]
        }
    ],
    "start": 1476321533
}

Other Approaches Considered: 

#2 Collapse=program: Programs are collapsed into workflows if applicable. Mapping of program to workflow is maintained in this case. This approach is an extension of the collapse approach used today. But since workflows are also programs, it has the ambiguity when collapsing on "programs" because it still shows workflows. 

• JSON Response Changes: 

1. In the "programs" section: the ProgramId of the Program will be replaced by ProgramId of the workflow if the program is associated with a workflow. If not the Programs show as it is.
2. In the "relations" section: a new field called "workflow" will be added for all the relations that could be collapsed based on workflows. The programs field will be a collection of all the programs that were collapsed to form this workflow.
   
   

#3 Group-by on Workflow: In addition to the collapse, if a group-by is provided, it can be used as group-by=workflow and the programs that dont have workflows associated are left as it is. This approach also has some level of ambiguity listed above but is extensible if in future there is a requirement for group-by on application. [Because collapse workflow is confusing if there are no workflows present]