...
As a result, the metadata system will have to manage two different datasets. The storage format of both datasets (both keys and values) will be identical, they will only write to separate tables.
A higher level construct (TBD, but an extended BusinessMetadataStore or MetadataAdmin) , MetadataStore will have to be extended the ability to interact with two separate datasets. It will use a MetadataScope (possible values USER and SYSTEM) object to distinguish between operations that should go to the business metadata dataset from the ones that should go to the system metadata dataset.
The MetadataStore class is chosen to have the ability to interact with two separate datasetsdifferent metadata datasets, because it is the API that is used across CDAP (LineageDatasetFramework, Lineage classes, StreamAdmin, AppLifecycleService, DeletedProgramHandlerStage, to name a few classes) to interact with Metadata. There was an option to have this ability in the MetadataAdmin object instead and have the MetadataStore be local to a specific dataset (this may have made the MetadataStore class itself cleaner). However, this way, we would have needed the downstream classes (users of MetadataStore) handle multiple MetadataStores, which is not clean. Also, currently, the MetadataAdmin is only used by the MetadataHttpHandler. As a result, we cannot move this logic to the MetadataAdmin class, since not all clients of the metadata system have access to it.
The MetadataAdmin class is currently in app-fabric, because it needs access to the AppMetadataStore to check if entities exist. This is not ideal, but to fix this, we need to split cdap-app-fabric, which is much beyond the scope of the Metadata work.
History
We will re-use the same pattern that the Business Metadata Dataset uses to store history.
...