...
Id | Description | Comments |
---|---|---|
U1 | As a user, I should be able to search Datasets containing the specified fields | List the kinds of queries that will be supported |
U2 | As a CDAP system, I should be able to annotate CDAP entities with system metadata automatically | System metadata for each entity is listed below |
U3 | As a user, I should be able to access and update CDAP metadata using the CDAP CLI | |
U4 | As a developer, I should be able to access and update CDAP metadata using the CDAP Test Framework | |
U5 | As a user, I should be able to search CDAP entities based on metadata using the CDAP UI | |
U6 | As a user, I should be able to view the lineage of a CDAP dataset/stream in a specified time window using the CDAP UI |
System Metadata
Kinds of system metadata:
...
For interacting with the System Metadata Dataset, we will introduce a SystemMetadataUpdater
interface, which will be injected at various stages outlined below, to add, update or delete business system metadata
System Metadata will be added when:
...
Purpose | API | Body | Response | Routable | Comments | Approved? | ||
---|---|---|---|---|---|---|---|---|
Annotate business metadata for view | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties |
| 200: Successful 404: view not found in specified namespace | Yes |
|
| ||
Retrieve business metadata for view | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties | N/A | 200: Successful 404: View not found in specified namespace
| Yes | ||||
Delete all business metadata for view | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Delete selected key from business metadata for view | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties/{key} |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Search views containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=view | N/A | 200: Successful
| Yes |
| |||
Add tags to a view | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags |
| 200: Successful 404: View not found in specified namespace | Yes |
|
| ||
Retrieve view tags | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags | N/A |
| Yes | ||||
Remove all view tags | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Remove specified view tag | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags/{tag} |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Get all business metadata for a view | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata |
| 200: Successful 404: View not found in specified namespace | Yes | Retrieves all properties and tags for a stream. |
Existing/Changed REST APIs and CLI Commands:
Note: Changes are in blue
Purpose | API | CLI Command | Body | Response | Comments | Approved? | ||
---|---|---|---|---|---|---|---|---|
Annotate business metadata for datasets | POST /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | set metadata properties datasets <dataset-id> |
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Annotate business metadata for apps | POST /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | set metadata properties apps <app-id> |
| 200: Successful 404: App not found in specified namespace |
|
| ||
Annotate business metadata for programs | POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | set metadata properties app <app-id> program-type <program-type> |
| 200: Successful 404: Program not found in specified namespace |
|
| ||
Annotate business metadata for streams | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | set metadata properties streams <stream-id> |
| 200: Successful 404: Stream not found in specified namespace |
|
| ||
Retrieve business metadata for datasets | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | get metadata properties scope datasets <dataset-id> | N/A | 200: Successful 404: Dataset not found in specified namespace
| ||||
Retrieve business metadata for apps | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | get metadata properties scope apps <app-id> | N/A | 200: Successful 404: App not found in specified namespace
| ||||
Retrieve business metadata for programs | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | get metadata properties scope apps <app-id> program-type <program-id> | N/A | 200: Successful 404: Program not found in specified namespace
| ||||
Retrieve business metadata for streams | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | get metadata properties scope streams <stream-id> | N/A | 200: Successful 404: Stream not found in specified namespace
| ||||
Delete all business metadata for datasets | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | delete metadata properties datasets <dataset-id> |
N/A | 200: Successful 404: Dataset not found in specified namespace | ||||
Delete selected key from business metadata for datasets | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties/{key} | delete metadata properties datasets <dataset-id> <key> |
N/A | 200: Successful 404: Dataset not found in specified namespace | ||||
Delete all business metadata for apps | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | delete metadata properties apps <app-id> |
| 200: Successful 404: App not found in specified namespace | ||||
Delete selected key from business metadata for apps | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties/{key} | delete metadata properties apps <app-id> <key> |
| 200: Successful 404: App not found in specified namespace | ||||
Delete all business metadata for programs | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | delete metadata properties apps <app-id> program-type <program-id> |
| 200: Successful 404: Program not found in specified namespace | ||||
Delete all business metadata for programs | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties/{key} | delete metadata properties apps <app-id> program-type <program-id> <key> |
| 200: Successful 404: Program not found in specified namespace | ||||
Delete all business metadata for streams | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | delete metadata properties streams <stream-id> |
| 200: Successful 404: Stream not found in specified namespace | ||||
Delete selected key from business metadata for streams | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties/{key} | delete metadata properties streams <stream-id> <key> |
| 200: Successful 404: Stream not found in specified namespace | ||||
Search Datasets entities containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=<target-type>
target-type => dataset, app, program, stream, view
| search metadata scope <search-query> <target> | N/A | 200: Successful
|
| |||
Apps containing business metadataView Dataset Lineage | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=app | search metadata scope <search-query> <target> | N/A | 200: Successful
|
| |||
Search Programs containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=program | search metadata scope <search-query> <target> | N/A | 200: Successful
|
| |||
Search Streams containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=stream | search metadata scope <search-query> <target> | N/A | 200: Successful
|
| |||
View Dataset Lineage | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/lineage?start=<start-ts>&end=<end-ts>&datasets/{dataset-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels> | get lineage datasets <dataset-id> <startTs> <endTs> <maxLevels> | N/A | 200: Successful Response TBD, but will contain a DAG representation | ||||
View Stream Lineage | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels> | get lineage streams <stream-id> <startTs> <endTs> <maxLevels> | N/A | 200: Successful Response TBD, but will contain a DAG representation | ||||
View Run Id Accesses | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/metadata | get metadata apps <app-id> program-type <program-id> runs <run-id> | N/A | 200: Successful Response Body TBD |
| |||
Add tags to a dataset | POST /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | add metadata tags datasets <dataset-id> |
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Add tags to an app | POST /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | add metadata tags apps <app-id> |
| 200: Successful 404: App not found in specified namespace |
|
| ||
Add tags to a program | POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | add metadata tags apps <app-id> program-type <program-id> |
| 200: Successful 404: Program not found in specified namespace |
|
| ||
Add tags to a stream | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags |
| 200: Successful 404: Stream not found in specified namespace |
|
| |||
Retrieve dataset tags | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | get metadata tags datasets <dataset-id> | N/A |
| ||||
Retrieve app tags | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | N/A |
| |||||
Retrieve program tags | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | N/A |
| |||||
Retrieve stream tags | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags | N/A |
| |||||
Remove all dataset tags | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | delete metadata tags datasets <dataset-id> | N/A
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Remove specified dataset tag | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: Dataset not found in specified namespace |
|
| |||
Remove all app tags | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | N/A
| 200: Successful 404: App not found in specified namespace | |||||
Remove specified app tag | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: App not found in specified namespace | |||||
Remove all program tags | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | N/A
| 200: Successful 404: Program not found in specified namespace | |||||
Remove specified program tag | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: Program not found in specified namespace | |||||
Remove all stream tags | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags |
| 200: Successful 404: Stream not found in specified namespace | |||||
Remove specified stream tag | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags/{tag} |
| 200: Successful 404: Stream not found in specified namespace | |||||
Remove all business metadata for a dataset | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata |
| 200: Successful 404: Dataset not found in specified namespace | Removes all properties and tags from a dataset. Will not happen in 3.2 | ||||
Remove all business metadata for an app | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata |
| 200: Successful 404: App not found in specified namespace | Removes all properties and tags from an app. Will not happen in 3.2 | ||||
Remove all business metadata for a program | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata |
| 200: Successful 404: Program not found in specified namespace | Removes all properties and tags from a program. Will not happen in 3.2 | ||||
Remove all business metadata for a dataset | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata |
| 200: Successful 404: Dataset not found in specified namespace | Removes all properties and tags from a dataset. Will not happen in 3.2 | ||||
Get all business metadata for a dataset | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata?scope=system/user |
| 200: Successful 404: Dataset not found in specified namespace | Retrieves all properties and tags for a dataset. Will not happen in 3.2 | ||||
Get all business metadata for an app | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata |
| 200: Successful 404: App not found in specified namespace | Retrieves all properties and tags for an app. Will not happen in 3.2 | ||||
Get all business metadata for a program | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata |
| 200: Successful 404: Program not found in specified namespace | Retrieves all properties and tags for a program. Will not happen in 3.2 | ||||
Get all business metadata for a stream | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata |
| 200: Successful 404: Stream not found in specified namespace | Retrieves all properties and tags for a stream. Will not happen in 3.2 |
...