Versions Compared

Old Version 11

changes.mady.by.user Gokul Gunasekaran

Saved on

compared with

New Version 12

changes.mady.by.user Gokul Gunasekaran

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

CDAP Metadata in Cloudera Navigator

...

  •  User stories documented (Gokul)
  •  User stories reviewed (Nitin)
  •  Design documented (Gokul)
  •  Design reviewed (Andreas)
  •  Feature merged (Gokul)
  •  Examples and guides (Gokul)
  •  Integration tests (Gokul) 
  •  Documentation for feature (Gokul)
  •  Blog post (Gokul)


User Stories: 

  • The user can query/modify business metadata using the Metadata API that CDAP provides. But if the user is already using Navigator for this purpose, he CDAP business and system metadata entities should automatically show up in Navigator (if Navigator is chosen in configuration)
  • CDAP user should be able to continue to do so for CDAP entities as well.
      

  • The user should be able view the System Metadata in Navigator and should also be able to modify Business Metadata. Also, parts of System Metadata should be indexed (for example, name of the entity) and be query-able 
    and all of Business Metadata should be editable and query-able through Navigator.

  • The users should be able to seamlessly switch between Navigator and CDAP to manage business/custom metadata for CDAP entitiessearch for CDAP business and system metadata using Cloudera navigator
  • Any updates/deletes to system or business metadata in CDAP should automatically reflect in Navigator
  • Users should be able to search on dataset or streams schema fields (fine-grained entities) in Navigator
  • Existing metadata (data that existed before Navigator integration was enabled) should also be made available in Navigator (depends on whether messages are available in Kafka) (Low priority)
  • Updates/deletion of custom metadata in Navigator should be reflected in CDAP (Low priority)

  • Advanced User Requirement: Pushing business metadata of CDAP entities to lower level systems - For example, if a CDAP Table dataset is marked as ‘sensitive’, this tag should be pushed to the corresponding HBase Table
    created by CDAP.    But the question of what are the correct levels of abstractions - PFS => HDFS directories + HBase table. No entities in other systems (HDFS, HBase) for Namespace (/cdap/namespaces/myspace), Programs? 
    This is an advanced requirement (requires platform support to figure out sub-entities for CDAP entities) and is not dealt with below. . (Low priority)     

Design:

Technical Constraints 

...