Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Overview

This page covers the requirements, design and implementation of metadata and data discovery features in 3.3

High Level Requirements

  1. Metadata search
  2. Schema as metadata
  3. System metadata
  4. CLI, Test Framework Support for metadata
  5. UI for Metadata Search
  6. UI for Lineage
  7. UI for Adding/Updating metadata properties/tags
  8. Lineage based on Type of Dataset Access
  9. Monitoring/Logs for Metadata Service

Scope

  1. Schema as metadata
  2. System metadata
  3. Metadata CLI
  4. Test Framework support for Metadata
  5. UI... (needs to be finalized)

User Stories

IdDescriptionRequirements FulfilledComments
U1As a user, I should be able to search Datasets containing the specified fields List the kinds of queries that will be supported
U2As a CDAP system, I should be able to annotate CDAP entities with system metadata automatically 

List all the system tags that should be annotated

  • Kind of entity (dataset, app, program, program type, stream)?
  • artifact name

 

U3As a user, I should be able to access and update CDAP metadata using the CDAP CLI  
U4As a developer, I should be able to access and update CDAP metadata using the CDAP Test Framework  
U5As a user, I should be able to search CDAP entities based on metadata using the CDAP UI  
U6As a user, I should be able to view the lineage of a CDAP dataset/stream in a specified time window using the CDAP UI  
    

 

System Metadata

Kinds of system metadata:

Applications

  • Artifact name

Programs

  • Type of program

Datasets

  • Type of dataset
  • Creation time - property
  • Last update time? - property
  • RecordScannable/BatchWritable/RecordWritable/BatchReadable
  • Other properties

Streams

  • Format

Schema as Metadata

Schema as metadata is meant to add the capability in CDAP for users to be able to retrieve datasets/streams with a field X optionally of type Y.

Design Considerations

Storage

There is a case for storing System Metadata in a separate dataset for the following reasons:

  1. Only the CDAP system can update System Metadata.  
  2. System Metadata may have different authorization as well as retention policies than Business Metadata
  3. System Metadata can be updated at specific times only, where users can update Business Metadata at any given time

However, if stored as a separate dataset, the metadata system will have to manage two different datasets. APIs may need filters, etc - TODO: Details

Runtime

System Metadata will be added/updated when:

  1. An app is deployed - We will add a SystemMetadataUpdater stage in the deployment pipeline that will update system metadata for the app, as well as all the programs in the app.
  2. A new dataset instance is created - The LineageWriterDatasetFramework can be extended to update system metadata when a dataset is added.
  3. A new stream is created - 

System Metadata Updates

Only the CDAP system can update system metadata for entities. This capability will not be exposed to users. However, given this design choice, users will need a capability in CDAP to discover all the system tags/properties. To start off with, this can be exposed via a simple API that lists all tags/properties. It can later be extended via full-text search capabilities when CDAP has a more comprehensive search capability that extends beyond IndexedTables and prefix lookups.

Questions

 

 

 

 

  • No labels