...
System Metadata will be added /updated when:
- An app is deployed - We will add a SystemMetadataUpdater stage in the deployment pipeline that will update system metadata for the app, as well as all the programs in the app.
- A new dataset instance is created - The LineageWriterDatasetFramework can be extended to update system metadata when a dataset is added.
- A new stream is created -
...
System Metadata will be updated when:
- A dataset instance's properties are updated
- A stream's config is updated
System Metadata will be deleted when:
- An app is deleted
- A program is removed from an existing app
- A dataset instance is deleted
- A stream is deleted
System Metadata Updates
Only the CDAP system can update system metadata for entities. This capability will not be exposed to users. However, given this design choice, users will need a capability in CDAP to discover all the system tags/properties. To start off with, this can be exposed via a simple API that lists all tags/properties. It can later be extended via full-text search capabilities when CDAP has a more comprehensive search capability that extends beyond IndexedTables and prefix lookups.
...