Versions Compared

Old Version 33

changes.mady.by.user Rohit Sinha

Saved on

compared with

New Version 34

changes.mady.by.user Rohit Sinha

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

IdDescriptionComments
U1As a user, I should be able to search Datasets containing the specified fieldsList the kinds of queries that will be supported
U2As a CDAP system, I should be able to annotate CDAP entities with system metadata automatically

System metadata for each entity is listed below

U3As a user, I should be able to access and update CDAP metadata using the CDAP CLI 
U4As a developer, I should be able to access and update CDAP metadata using the CDAP Test Framework 
U5As a user, I should be able to search CDAP entities based on metadata using the CDAP UI 
U6As a user, I should be able to view the lineage of a CDAP dataset/stream in a specified time window using the CDAP UI 

 

New Metadata Search:

Metadata Storage and Search to support:

  1. Key-Value Metadata:
    1. Codename: Alpha Tango Charlie
      Supported searches:
      1. Whole Key-Value

...

Tags: Tag1, Tag2

...

      1. (complete or partial) - Codename: Alpha Tango Charlie or Codename: Alpha Tang*
      2. Key with Part of Value (complete or partial) - Codename: Alpha or Codename: Tango or Codename: Charlie or Codename: Alp*
      3. Whole Value (complete or partial): Alpha Tango Charlie or Alpha* or Alpha Tan*
      4. Parts of value (complete or partial): Alpha or Tango or Charlie or Alph* or Tan* or Ch*

  2. Tags Metadata:
    1. Tags: Tag1, Tag22
      Supported searches:
      1. With tags key and a tag value (complete or partial): Tags: Tag1 or Tags: Tag*
      2. With tag value complete or partial: Tag22 or  Tag2*

  3. Schema Metadata: This is just key-value where key is schema and value schema fields but needs special indexing to support searches with fieldName and fieldName:fieldType.
    1. Schema: {EmpName: String, EmpContact: {EmpTel: Integer, EmpAddr: String}} (Note: This is a nested schema)
      Supported searches:
      1. FieldName with FieldType: EmpName: String or EmpTel:Integer or EmpAddr: String
      2. FieldName: EmpName, EmpTel, EmpAddr

 

 

Storage:

Real Table

...

Main Table: This table stores stores the metadata for the entity. It will be used when an user wants to get metadata of an entity. This table is not for searching. 

Key: Entity with keyValue: Value of Metadata
<Entity-Id><CodeName>Alpha Tango Charlie
<Entity-Id><Tags>{Tag1, Tag2Tag22}
<Entity-Id><Schema-Id>{field1: fieldtype1, field2: fieldtype2{EmpName: String, EmpContact: {EmpTel: Integer, EmpAddr: String}}

 

Index Table: This table will be used for searching and it will use IndexedTable.

Key: Entity with indexValue:

...

<Entity-Id>
IndexesEntity
Author: Rohit Sinha<Entity-Id>
Author: Rohit<Entity-Id>
Author: Sinha<Entity-Id>
Rohit Sinha<Entity-Id>
Rohit<Entity-Id>
Sinha<Entity-Id>
Tag1<Entity-Id>
Schema<Entity-Id>
field1:fieldtype1<Entity-Id>
field1<Entity-Id>
field2:fieldtype2<Entity-Id>
field2Index
<Entity-Id><CodeName: Alpha Tango Charlie>CodeName: Alpha Tango Charlie
<Entity-Id><Codename: Alpha>Codename: Alpha
<Entity-Id><Codename: Tango>Codename: Tango
<Entity-Id><Codename: Charlie>Codename: Charlie
<Entity-Id><Alpha Tango Charlie>Alpha Tango Charlie
<Entity-Id><Alpha>Alpha
<Entity-Id><Tango>Tango
<Entity-Id><Charlie>Charlie
<Entity-Id><Tags: Tag1>Tags: Tag1
<Entity-Id><Tags: Tag22>Tags: Tag22
<Entity-Id><Tag1>Tag1
<Entity-Id><Tag22>Tag22
  
  
  

 

 

 

System Metadata

Kinds of system metadata:

...