Page Comparison

...

1. User should be able to search key-value metadata with the following or with its prefix:
   • key-value 
   • key with part of value
   • value
   • Individual words in the value
     
     Example:
     User stores a key-value metadata with key = "Codename" and value = "Alpha Tango Charlie" for an entity
   • User should be able to search for this entity with the following queries:
     • key-value
       1. Codename: Alpha Tango Charlie
       2. Codename: Alpha Tang*
     • key with part of value
       1. Codename: Alpha
       2. Codename: Tango
       3. Codename: Charlie
       4. Codename: Alp*
     • value
       1. Alpha Tango Charlie
       2. Alpha*
       3. Alpha Tan*
          Design DecisionNote:
          1. We have decided not to support searches for queries which have parts of value for example "Tango Charlie". You can either search for whole value or with prefix or single words (we plan to tokenize on whitespace) 
     • parts of value
       1. Alpha
       2. Tango
       3. Charlie
       4. Alph*
       5. Tan*
       6. Ch*
2. User should be able to search tags metadata with the following or with its prefix:
   • tags key and a tag value
   • a tag value
     
     Example:
     User tags an entity with the following tags "Tag1, Tag22"
   • User should be able to search for this entity with the following queries:
     • tag key and a tag value:
       1. tags: Tag1
       2. tags: Tag*
     • a tag value
       1. tag22 
       2. tag2*

======================================================================================================================

Use Cases:

1. Key-Value Metadata:Codename: Alpha Tango Charlie
   Use case:
2. User should be able to search with
   
3. Whole Key-Value (complete or prefix) 
   1. Codename: Alpha Tango Charlie
   2. Codename: Alpha Tang*
4. Key with Part of Value (complete or prefix) 
   1. Codename: Alpha
   2. Codename: Tango
   3. Codename: Charlie
   4. Codename: Alp*
5. Whole Value (complete or prefix): 
   1. Alpha Tango Charlie
   2. Alpha*
   3. Alpha Tan*
      Design Decision:
      1. We have decided not to support searches for queries which have parts of value for example "Tango Charlie". You can either search for whole value or with prefix or single words (we plan to tokenize on whitespace) 
Parts of value (complete or prefix): 
6. Alpha
7. Tango
8. Charlie
9. Alph*
10. Tan*
Ch*
Tags Metadata:tags: Tag1, Tag22
Use case: search for entities (datasets, streams, views) through field-names in schema with the following or with its prefix:
1. fieldname
2. fieldname scoped with schema - this should limit the search to just schema fields and not other metadata

3. User should be able to search

   with
   
4. With tags key and a tag value (complete or prefix): 
   1. tags: Tag1
   2. tags: Tag*
With tag value (complete or prefix): 
5. tag22 
tag2*

Schema Metadata: This is just key-value where key is schema and value schema fields but needs special indexing to support searches with fieldName (Note: we don't plan to support schema searches with fieldTypes

1. for all entities with which has a schema
   
   Example:
   A dataset has the following schema: 

   Code Block
   title Nested Schema
   { "EmpName": "String", "EmpContact": { "EmpTel": "Integer", "EmpAddr": "String" } }

   Use case: User should be able to search for this dataset with

   FieldName

   the following queries:

   • fieldname:
     1. EmpName
     2. EmpContact
     3. EmpTel 
     4. EmpAddr
     5. Emp*
   • fieldname scoped to schema
   (complete or prefix)
   • :
    Searching for everything which has
   • 1. schema: EmpName
     2. schema: EmpContact
     3. schema: EmpTel
     4. schema: EmpAddr 
     5. schema: Emp*
2. FieldName (complete or prefix): 
   1. EmpName
   2. EmpContact
   3. EmpTel 
   4. EmpAddr
   5. Emp*
• search for all entities with a schema
  1. schema:*

Design Decisions
• 1. This will return this dataset entity and also all the other entities which have schema stored as their metadata
     
     
Note:

• We don't plan to support schema searches with fieldType.
•  If a user  searched with a query which is not scoped with schema by default it will search for schema fields besides the normal key-value and tags.
  Open questions:
  • What if an entity has multiple schema (ex: transform which has input and output schema)
    • Maybe We can index its fields with input and output schema and we expect an user to specify whether they are looking for something in input schema or output schema. 
  • What about entities which have more than one schema?
   We are thinking to
  • • • Maybe we can store them either as input output with identifier.
    • How will an user search for a fieldName across input and output schema ?
      • One way is to besides indexing the fields as input and output schema we also index every field as just schema so that we can perform such queries.
        

New Design:

Storage:

Metadata Storage Format:

...