Versions Compared

Old Version 1

changes.mady.by.user Todd Greenstein

Saved on

compared with

New Version Current

changes.mady.by.user Todd Greenstein

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...

...

...

...

...

...

...

...

...

Goals

  • The app store market will provide an application store for CDAP users to quickly and efficiently add capabilities into the platform.   
  • The app store will also expand capabilities within the community by allowing users to publish capabilities for other CDAP users to experience.  Cask Market will be a read only for 4.0 and include provisions for working offline.   

Checklist

  •  User stories documented(Todd)
  •  Requirements documented(Todd)
  •  Requirements Reviewed(Nitin, Jon, Russ, Sree, Ajai)
  •  Mockups Built(Nitin, Todd)
  •  Design Built(Sean)
  •  Design Accepted(Nitin, Jon, Russ, Ajai)

User Stories

  • As a CDAP User I want a plus button on the navbar so that I can an add new capabilities from anywhere in the platformAs a CDAP User I want to be able to add new Datasets from the application store with pre configured fields so that I can easily write applications with schemas already defined for popular technologies, like omniture.
  • As a CDAP User I want to be able to add new artifacts from the app store market into the system to speed up development and improve the overall developer experience.
  • As a Hydrator User I want to be able to add plugins from the app store market to expand the capabilities of my pipeline.  
  • As a Hydrator User from within the Hydrator Studio I want to be able to add new pipelines or applications from within Hydrator, and upon adding, remain in the Hydrator context.
  • As a CDAP user I want to be able to add examples from the app storemarket.  
  • As a CDAP user from within the app store market I want to be able to know what I've added into the platform and have those special cased when browsing the app storemarket.  
  • AS a CDAP user I want to have access to the market in offline mode, so that I can add capabilities into the system from where-ever my cluster is deployed.  

Use Cases

Scenario 1
[1] I'm in hydrator studio, in CDAP 4.0
[2] I click the + button
[3] From the

...

market I choose pipeline, and add "SFDC Lead dump" 
[4]

...

The wizard walks me through the rep defined steps to add the pipeline
[5] I return to the hydrator context, and it's in the pipeline studio, AND NOT deployed as an app
Scenario 2
[1] In CDAP, and I click the + button
[2] From the

...

market I choose pipeline, and add "SFDC Lead dump" 
[3] The wizard walks me through the repo defined steps to add the pipeline
[4] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 3
[1] In CDAP, and I click the + button
[2] From the

...

market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4] If I click "clone in hydrator"

...

the wizard walks me through the repo defined steps to add the pipeline
[5] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 4
[1] In CDAP, and I click the + button
[2] From the

...

market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4]

...

 If I click "add", the wizard walks me through the repo defined steps to deploy the application
[5] The app will be deployed and I will see in detailed pipeline view.  
Scenario 5
[1] In CDAP, and I click the + button
[2] From the

...

market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added as a resource, and the resource center shows this as included, with the specific entity version.
Scenario 6
[1] In hydrator studio, and I click the + button
[2] From the

...

market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added and will show up on in the plugin library for immediate use
Scenario 7

[1] In hydrator studio, and I click the + button
[2] From the market I choose add a third party plugin, "MYSQL JDBC"
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the plugin.
[5] In the case of an external dependency, the license agreement and download should be incorporated into the wizard.
[6] The option to have it added as plugin template should be part of the wizard.
[7] The plugin is added and will show up on in the plugin library for immediate use.

Scenario 8

[1] In CDAP, and I click the + button
[2] From the market I choose add "Traffic Incidents"  mysql data 
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the datapack locally and into my development environment 

Scenario 8 Workflow Diagram

Image Added

Requirements

Market Requirements

  • The app store market should be accessible from anywhere within CDAP.
  • The app store market should support adding:
      Datasets
    • Artifacts
      • Applications
      (including custom applications/examples)
      • Native Plugins
      • Third Party Plugins
    • Pipelines
    • PluginsDataPacks
  • The app store market should provide context when browsing to know if I've already added an entity.  The app store should special case if a newer version of an entity I added is available I should still be able to create another instance and the market should keep track.  
  • The app store market should instantiate the new entity wizard where applicable.  
  • The app store should have rules for contribution and allow user submissions.The app store market should be browsable independently of the product, from the website.  
  • The app store market on the website should link to localhost and the cdap port to indicate "open in cdap" to add entities.  
  • The app store market should only be supported by default in both Distributed and Standalone, with the option to be disabled.  
  • The market should work in offline mode after syncing the repository onto the cluster.  
  • The market should be an independent entity outside of CDAP.
  • CDAP should track what has been added previously
  • The market should be read only for the first iteration.   

Market Repository Requirements

  • The repo should include metadata only.
  • The contract between the repo and the entities needs to be versioned to support changes in the SDK. repo behavior or meta data specifications for entities.
  • The repo should be updatable/installable from the UI, REST API or CMD Line.  
  • The repo should include versioning information for entities.
  • The repo should include a checksum for entities.
  • The repo should include information for third party entities.
  • The repo should include author and licensing information for entities.
  • The repo should include dependency information for entities
  • The repo should include abstraction information (service descriptor) for how an entity will interact with the market wizard workflow.   The wizard will consist of 1 to N steps and must be able to abstract all information necessary from the entity descriptor for how an entity is added including dependencies (internal/external) precedence of steps.

Market Datapack Requirements

Datapacks are sources of data that will improve the bootstrapping and getting started experience on the platform.  

  • Datapacks should be able to be added for:
    • SQL datapacks 
      • MYSQL .sql files
      • SQLITE .sqlite
    • S3 datapacks
      • Text files
      • csv examples
    • HTTP (for hydrator realtime pipelines)
      • Endpoints for various supported web services
  • Datapacks should be available to be added as new sources plugin templates in hydrator.  
  • Datapacks should be able available for custom applications.