...
Id | Description | Comments |
---|---|---|
U1 | As a user, I should be able to search Datasets containing the specified fields | List the kinds of queries that will be supported |
U2 | As a CDAP system, I should be able to annotate CDAP entities with system metadata automatically | System metadata for each entity is listed below |
U3 | As a user, I should be able to access and update CDAP metadata using the CDAP CLI | |
U4 | As a developer, I should be able to access and update CDAP metadata using the CDAP Test Framework | |
U5 | As a user, I should be able to search CDAP entities based on metadata using the CDAP UI | |
U6 | As a user, I should be able to view the lineage of a CDAP dataset/stream in a specified time window using the CDAP UI |
System Metadata
Kinds of system metadata:
...
For interacting with the System Metadata Dataset, we will introduce a SystemMetadataUpdater
interface, which will be injected at various stages outlined below, to add, update or delete business system metadata
System Metadata will be added when:
...
The BusinessMetadataDataset dataset type introduced in 3.2 will be renamed to MetadataDataset, since it will also serve system metadata in 3.3. For existing CDAP installations, we will need an upgrade step to change the type of the existing "business.metadata" dataset in the "datasets.instance" table.
Jira Legacy | ||||||||
---|---|---|---|---|---|---|---|---|
|
New REST APIs
- The Metadata REST APIs to retrieve properties and tags will be updated to accept a scope query parameter. It will support the values user and system. If scope is not specified, the API will return all metadata across both scopes.
- New APIs will be added for View and artifacts:
Purpose | API | Body | Response | Routable | Comments | Approved? | ||
---|---|---|---|---|---|---|---|---|
Annotate business metadata for view | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties |
| 200: Successful 404: view not found in specified namespace | Yes |
|
| ||
Retrieve business metadata for view | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties | N/A | 200: Successful 404: View not found in specified namespace
| Yes | ||||
Delete all business metadata for view | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Delete selected key from business metadata for view | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/properties/{key} |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Search views containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=view | N/A | 200: Successful
| Yes |
| |||
Add tags to a view | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags |
| 200: Successful 404: View not found in specified namespace | Yes |
|
| ||
Retrieve view tags | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags | N/A |
| Yes | ||||
Remove all view tags | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Remove specified view tag | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata/tags/{tag} |
| 200: Successful 404: View not found in specified namespace | Yes | ||||
Get all business metadata for a view | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/views/{view-id}/metadata |
| 200: Successful 404: View not found in specified namespace | Yes | Retrieves all properties and tags for a stream. |
Existing/Changed REST APIs and CLI Commands:
Note: Changes are in blue
Purpose | API | CLI Command | Body | Response | Comments | Approved? | ||
---|---|---|---|---|---|---|---|---|
Annotate business metadata for datasets | POST /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | set metadata properties datasets <dataset-id> |
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Annotate business metadata for apps | POST /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | set metadata properties apps <app-id> |
| 200: Successful 404: App not found in specified namespace |
|
| ||
Annotate business metadata for programs | POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | set metadata properties app <app-id> program-type <program-type> |
| 200: Successful 404: Program not found in specified namespace |
|
| ||
Annotate business metadata for streams | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | set metadata properties streams <stream-id> |
| 200: Successful 404: Stream not found in specified namespace |
|
| ||
Retrieve business metadata for datasets | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | get metadata properties scope datasets <dataset-id> | N/A | 200: Successful 404: Dataset not found in specified namespace
| ||||
Retrieve business metadata for apps | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | get metadata properties scope apps <app-id> | N/A | 200: Successful 404: App not found in specified namespace
| ||||
Retrieve business metadata for programs | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | get metadata properties scope apps <app-id> program-type <program-id> | N/A | 200: Successful 404: Program not found in specified namespace
| ||||
Retrieve business metadata for streams | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | get metadata properties scope streams <stream-id> | N/A | 200: Successful 404: Stream not found in specified namespace
| ||||
Delete all business metadata for datasets | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties | delete metadata properties datasets <dataset-id> |
N/A | 200: Successful 404: Dataset not found in specified namespace | ||||
Delete selected key from business metadata for datasets | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/properties/{key} | delete metadata properties datasets <dataset-id> <key> |
N/A | 200: Successful 404: Dataset not found in specified namespace | ||||
Delete all business metadata for apps | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties | delete metadata properties apps <app-id> |
| 200: Successful 404: App not found in specified namespace | ||||
Delete selected key from business metadata for apps | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/properties/{key} | delete metadata properties apps <app-id> <key> |
| 200: Successful 404: App not found in specified namespace | ||||
Delete all business metadata for programs | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties | delete metadata properties apps <app-id> program-type <program-id> |
| 200: Successful 404: Program not found in specified namespace | ||||
Delete all business metadata for programs | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/properties/{key} | delete metadata properties apps <app-id> program-type <program-id> <key> |
| 200: Successful 404: Program not found in specified namespace | ||||
Delete all business metadata for streams | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties | delete metadata properties streams <stream-id> |
| 200: Successful 404: Stream not found in specified namespace | ||||
Delete selected key from business metadata for streams | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties/{key} | delete metadata properties streams <stream-id> <key> |
| 200: Successful 404: Stream not found in specified namespace | ||||
Search entities containing business metadata | GET /v3/namespaces/{namespace-id}/metadata/search?query=term&target=<target-type>
target-type => dataset, app, program, stream, view
| search metadata scope <search-query> <target> | N/A | 200: Successful
|
| |||
View Dataset Lineage | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels> | get lineage datasets <dataset-id> <startTs> <endTs> <maxLevels> | N/A | 200: Successful Response TBD, but will contain a DAG representation | ||||
View Stream Lineage | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/lineage?start=<start-ts>&end=<end-ts>&maxLevels=<max-levels> | get lineage streams <stream-id> <startTs> <endTs> <maxLevels> | N/A | 200: Successful Response TBD, but will contain a DAG representation | ||||
View Run Id Accesses | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/metadata | get metadata apps <app-id> program-type <program-id> runs <run-id> | N/A | 200: Successful Response Body TBD |
| |||
Add tags to a dataset | POST /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | add metadata tags datasets <dataset-id> |
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Add tags to an app | POST /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | add metadata tags apps <app-id> |
| 200: Successful 404: App not found in specified namespace |
|
| ||
Add tags to a program | POST /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | add metadata tags apps <app-id> program-type <program-id> |
| 200: Successful 404: Program not found in specified namespace |
|
| ||
Add tags to a stream | POST /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags |
| 200: Successful 404: Stream not found in specified namespace |
|
| |||
Retrieve dataset tags | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | get metadata tags datasets <dataset-id> | N/A |
| ||||
Retrieve app tags | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | N/A |
| |||||
Retrieve program tags | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | N/A |
| |||||
Retrieve stream tags | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags | N/A |
| |||||
Remove all dataset tags | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags | delete metadata tags datasets <dataset-id> | N/A
| 200: Successful 404: Dataset not found in specified namespace |
|
| ||
Remove specified dataset tag | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: Dataset not found in specified namespace |
|
| |||
Remove all app tags | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags | N/A
| 200: Successful 404: App not found in specified namespace | |||||
Remove specified app tag | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: App not found in specified namespace | |||||
Remove all program tags | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags | N/A
| 200: Successful 404: Program not found in specified namespace | |||||
Remove specified program tag | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata/tags/{tag} | N/A
| 200: Successful 404: Program not found in specified namespace | |||||
Remove all stream tags | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags |
| 200: Successful 404: Stream not found in specified namespace | |||||
Remove specified stream tag | DELETE /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags/{tag} |
| 200: Successful 404: Stream not found in specified namespace | |||||
Remove all business metadata for a dataset | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata |
| 200: Successful 404: Dataset not found in specified namespace | Removes all properties and tags from a dataset. Will not happen in 3.2 | ||||
Remove all business metadata for an app | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/metadata |
| 200: Successful 404: App not found in specified namespace | Removes all properties and tags from an app. Will not happen in 3.2 | ||||
Remove all business metadata for a program | DELETE /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata |
| 200: Successful 404: Program not found in specified namespace | Removes all properties and tags from a program. Will not happen in 3.2 | ||||
Remove all business metadata for a dataset | DELETE /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata |
| 200: Successful 404: Dataset not found in specified namespace | Removes all properties and tags from a dataset. Will not happen in 3.2 | ||||
Get all business metadata for a dataset | GET /v3/namespaces/{namespace-id}/datasets/{dataset-id}/metadata?scope=system/user |
| 200: Successful 404: Dataset not found in specified namespace | Retrieves all properties and tags for a dataset. Will not happen in 3.2 | ||||
Get all business metadata for an app | GET /v3/namespaces/{namespace-id}/apps/{app-id}/metadata |
| 200: Successful 404: App not found in specified namespace | Retrieves all properties and tags for an app. Will not happen in 3.2 | ||||
Get all business metadata for a program | GET /v3/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/metadata |
| 200: Successful 404: Program not found in specified namespace | Retrieves all properties and tags for a program. Will not happen in 3.2 | ||||
Get all business metadata for a stream | GET /v3/namespaces/{namespace-id}/streams/{stream-id}/metadata |
| 200: Successful 404: Stream not found in specified namespace | Retrieves all properties and tags for a stream. Will not happen in 3.2 |
Questions
- The REST APIs to retrieve metadata will accept an additional scope parameter. Is it considered a backward incompatible change that if the scope is not specified, the API will now return all metadata, and not just business metadata, like it did in 3.3?
...