...

It is virtually impossible to list all possible scenarios, but it is important to realize that any combination of the above scenarios must work seamlessly. For example, a dataset may be maintained by multiple apps, and still shared with many others. Or a dataset may be created through a Hydrator pipeline but shared with many other pipelines or apps. That also means that the simplest of use cases (Scenario 1) must be interoperable with the most complex one (Scenario 3). Also, any time there is a conflict between different apps, pipelines, plugins, or app store artifacts that attempt to create the same dataset, but with different types, or with a version conflict, etc., this conflict must be detected by CDAP and reported back to the user in a clear and easy-to-read way.

User Stories

[DTM] Dataset Type Management

1. As an app developer, I want to include the code of a dataset type in my app artifact, and create a dataset of that type when deploying the app.
2. As an app developer, I want to deploy a new version of a dataset type as part of deploying a new version of the app that includes it, and I expect that all dataset instances of that type that were created as part of the app deployment start using the new code. 
3. As an app developer, I want to share a dataset type that I had previously deployed as part of an app.
4. As an app developer, I want to deploy a new version of a dataset type as part of an app artifact, without affecting other datasets of this type.
5. As an app developer, I want to explore a dataset instance of a type that was deployed as part of an app.
6. As an app developer, I expect that deploying an artifact without creating an app will not create any dataset types or instances (that is, this only happens when creating an app).
7. As an app developer, I want to share a dataset type across multiple applications that include the dataset type's code in their artifacts.
8. As an app developer, when deploying a new version of an app that includes a shared dataset type, I expect that all dataset instances created by this app start using the new code, but all dataset instances created by other apps remain unchanged.
9. As an app developer, I want to deploy a new version of an app that includes an older version of a dataset type deployed by another app, and I expect that the dataset instances created by this app use the dataset type code included in this app.
10. As an app developer, when I deploy a new version of an app that includes an different version of a dataset type deployed by another app, and this app shares a dataset instance of this type with the other app, the deployment will fail with a version conflict error. (Because otherwise I might "downgrade" the instance to an older version, making it incompatible with the other app). 
    Note: This use case needs discussion. What is proper behavior? How can be prevent data corruption due to unintentional "downgrade" without restricting ease of use too much?
11. As a dataset developer, I want to deploy a dataset type independent from any app, and allow apps to create and use dataset instances of that type.
12. As a dataset developer, I want to separate the interface from the implementation of a dataset type.
13. As an app developer, I want to only depend on the interface of a dataset type in my app, and have the system inject the implementation at runtime. 
14. As an app developer, I want to write unit tests for a an app that depends on the interface of a dataset type. (This means I need an extra dependency with test scope in my pom.xml)
15. As a dataset developer, I want to assign explicit versions to the code of a dataset type.
16. As a dataset developer, I want to deploy a new version of a dataset type without affecting the dataset instances of that type.
17. As an app developer, I want to create a dataset instance with a specific version of a dataset type. 
18. As a dataset developer, I want to have the option of implementing an "upgrade step" for when a dataset instance is upgraded to a new version of the dataset type.
19. As a dataset developer, I want to have a way to reject an upgrade of a dataset instance to a newer version of it type, if the upgrade is not compatible. 
20. As a dataset developer, I want to have the option of implementing a migration procedure that can be run after an upgrade of a dataset instance to a new version of it type. This can be a long-running (background) process.
21. As a dataset developer, I want to implement custom administrative operations (such as "compaction", or "rebalance") that are no common to all dataset types.
22. As an app developer, I want to perform custom administrative operations on dataset instances from my app, the CLI, REST, or the UI. 
23. As a dataset developer, I want to explore a dataset instance created from a dataset type that was deployed by itself. 
24. As a dataset developer, I want to delete outdated versions of a dataset type. I expect this to fail if there are any dataset instances with that version of the type. 
25. As a dataset developer, I want to list all dataset instances that use a dataset type, or a specific version of a type.
26. As a data scientist or app developer, I want to be able to create a dataset instance of an existing dataset type without writing code.
27. As a data scientist or app developer, I want to be able to upgrade a dataset instance to a new version of its code.
28. As a hydrator user, I want to create a pipeline that reads or writes an existing dataset instance.
29. As a hydrator user, I want to create a pipeline that reads or writes a new dataset instance, and I want to create that dataset instance as part of pipeline creation. 
30. As a hydrator user, I want to specify an explicit version of the dataset types of the dataset instances created by my pipeline, and I expect pipeline creation to fail (similar to app creation) if that results in incompatible upgrade of an existing dataset instance that is shared with other apps or pipelines.
31. As a hydrator user, I want to explore the datasets created by my pipeline.
32. As a hydrator user, I expect all dataset instances created by apps to be available as sinks and sources for pipelines (if there is a corresponding plugin).
33. As an app developer, I expect all dataset instances created by Hydrator pipelines to be accessible to the app.
34. As a plugin developer, I want to include the code for a dataset type in the plugin artifact. When a pipeline using this plugin is created, a dataset instance of that type is created, and it is explorable and available to apps.
35. As a plugin developer, I want to use a custom dataset type (that was deployed independently or as part of an app) inside the plugin. 
36. As a plugin developer, I want to upgrade the code of a dataset type used by a dataset instance created by that plugin, when I deploy a new version of the plugin and update the pipeline to use that version.
37. As a pipeline developer, I want to upgrade a dataset instance to a newer version of the code after the pipeline was created. 

[DIC] Dataset Instance Configuration

[Note: "As a user" refers to app developers, data scientists, dev-ops, or Hydrator users, pipeline developers]

1. As a user, when creating a dataset instance, I want to find out what properties are supported by the dataset type, what values are allowed, and what the defaults are. 
2. As a user, I want to specify the schema of a dataset in a uniform way across all dataset types.
3. As a user, I want to specify schema as a JSON string (verbose, Avro-style).
4. As a user, I want to specify schema as a SQL schema string (brief, Hive-style).
5. As a user, I want to configure time-to-live (TTL) in a uniform way across all dataset types. 
6. As a user, I want to see the properties that were used to configure a dataset instance.
7. As a user, I want to find out what properties of a dataset can be updated.  
8. As a user, I want to update the properties of a dataset instance. I expect this to fail if the new properties are not compatible, with a meaningful error message.
9. As a user, I want to update a single property of a dataset instance, without knowing all other properties. For example, set the TTL without having to know the schema. 
10. As a user, I want to remove a single property of a dataset instance, without knowing all other properties. For example, remove the TTL without having to know the schema. 
11. As a user, I want to trigger a migration process for a dataset if updating its properties requires that.
12. As a user, I expect that if reconfiguration of a dataset fails, then no changes have taken effect. In other words, all steps required to reconfigure a dataset must be done as a single atomic action.
13. As an app developer, I expect that application creation fails if any of its datasets cannot be created.
14. As an app developer, I expect that application redeployment fails if any of its datasets cannot be reconfigured (if the new app spec specifies different configuration). 
15. As an app developer, when creating a dataset as part of app deployment, I want to tolerate existing datasets if their properties are different but compatible. For example, I can configure the dataset schema, but leave the existing TTL of a table untouched.
16. As a pipeline designer, I want to use an existing dataset as a sink or source. If the schema (or any other property) of the dataset is incompatible with what the pipeline requires, I expect that pipeline creation fails with a meaningful error message. 

[EI] Explore Integration

1. As a user, I want to specify as part of dataset configuration whether it is explorable.
2. As a user, I do not want to specify the explore schema (and format) as separate properties if they can be derived from other standard dataset properties.
3. As a user, I want to specify the explore schema separately (for example, only include a subset of the fields of a table, or name fields differently).
4. As a user, I expect that dataset creation fails if the dataset cannot be enabled for explore.
5. As a user, I expect that dataset reconfiguration fails if the corresponding update of the explore table fails.
6. As a user, I expect that a dataset operation fails if it fails to make its required changes to explore.
7. As a user, I expect that an update of explore never leads to silent loss of data (or data available for explore). If, for example, partitions would be dropped from the explore table, I want to have the option to either fails the update, or to be notified of the drop and have a tool to bring explore in sync with the data. 
8. As a user, I want to enable explore for a dataset that was not configured for explore initially.
9. As a user, I want to disable explore for a dataset that was configure for explore initially.

...