Goal

In CDAP 4.0, the main theme for Datasets is improving/establishing proper and semantically sound dataset management. That includes the management of dataset types (code), and the management of dataset instances (actual data) throughout their life cycle. The current dataset framework has various shortcomings that need to be addressed. This document will discuss each area of improvement, list end-to-end use cases and requirements, and finally address the design to implement the requirements.

...

•  User stories documented(Andreas)
•  User stories reviewed(Nitin)
•  User stories reviewed(Todd)
•  Requirements documented(Andreas)
•  Requirements Reviewed
•  Mockups Built
•  Design Built
•  Design Accepted

...

1. As a user, I want to specify as part of dataset configuration whether it is explorable.
2. As a user, I do not want to specify the explore schema (and format) as separate properties if they can be derived from other standard dataset properties.
3. As a user, I want to specify the explore schema separately (for example, only include a subset of the fields of a table, or name fields differently).
4. As a user, I expect that dataset creation fails if the dataset cannot be enabled for explore.
5. As a user, I expect that dataset reconfiguration fails if the corresponding update of the explore table fails.
6. As a user, I expect that a dataset operation fails if it fails to make its required changes to explore.
7. As a user, I expect that an update of explore never leads to silent loss of data (or data available for explore). If, for example, partitions would be dropped from the explore table, I want to have the option to either cancel the update, or to be notified of the drop and have a tool to bring explore in sync with the data. 
8. As a user, I want to enable explore for a dataset that was not configured for explore initially.
9. As a user, I want to disable explore for a dataset that was configure for explore initially.

...

Requirements

[DTM] Dataset Type Management

1. Unification of artifact management for plugins, apps and dataset types
   1. Explicit versioning of dataset types
   2. Coexistence of multiple versions of the same dataset type
   3. Explicit dependency of a dataset instance on a specific version of its type
   4. Explicit upgrade of a dataset instance to a new version of its type
2. Preserve the easy experience of self-contained apps using an implicit versioning scheme
   1. Apps can bundle dataset types, create dataset instance of that type
   2. At runtime, such dataset types are loaded from the program jar
   3. Seamless experience when redeploying such an app
3. Dataset Admin needs a new method to upgrade to a new version of the type
   1. This method can reject the upgrade
4. Hydrator Plugins can also contain dataset types
5. Ability to take a dataset instance offline for a migration procedure
6. Injection of dataset code at runtime
   1. Not for dataset types embedded in app artifact (see 2.)
   2. Always inject the version of the type that the dataset instance is tagged with
   3. No noticeable performance degradation (some degradation is expected due to code injection at first instantiation of a type)
7. Backward-compatibility with existing dataset modules
8. Versioning for system dataset types
   1. Core types always use latest system version
   2. Composite types: TBD
9. New dataset admin API for performing custom actions
10. New versioned REST and CLI methods for versioned type and instance management
11. Maven archetype for dataset artifacts

[DIC] Dataset Instance Configuration

[EI] Explore Integration

1. Simplification of explore configuration
   1. Whether explore is enabled is explicit property
   2. All other explore properties derived from dataset properties if possible 
2. Explore failure also fails the DTM operation that called it
3. Ability to communicate warnings to the user for successful explore operations
4. Enable/Disable explore as dataset management operations