Hydrator Backend Application

To develop a back-end app to encapsulate business logic, that acts as as an intermediary between CDAP-UI and CDAP backend. The back-end app simplifies developing new features in CDAP-UI as it encapsulates the logic to translate business logic request/action to appropriate CDAP backend requests/actions and returns to the UI relevant information. This will make CDAP-UI to focus more on the UI aspects and less about the business logic involved.  Ideally this back-end app will remove the "view in CDAP" as the UI will be able to get the relevant information required from the backend-app.

Checklist

• User stories documented (Shankar)
• User stories reviewed (Nitin)
• Design documented (Shankar)
• Design reviewed (Terence/Andreas/Albert)
• Feature merged (Shankar)
• UI updated (Ajai/Edwin)
• Documentation for feature (Shankar)

Use-cases

Case #1

• User adds a database plugin to the pipeline, clicks on the database plugin to fill in the configuration
• User provides JDBC string, table name or SELECT query, username, password.
• User then clicks on the button to populate the schema
• UI will make the backend call to Hydrator App to retrieve the schema associated depending on whether it's based on Table or SELECT query.
• User then has the choice to include the schema as the output schema of the database plugin.
• The information of the schema associated with the database plugin is stored as spec in the exported pipeline. 

Case #2

• User adds a database plugin to the pipeline, clicks on the database plugin to fill in the configuration
• User provides JDBC string (include database and other configurations), username and password
• User on selecting table will click on the button to list the tables. 
• UI makes the backend call to retrieve the list of tables and show it to the user
• User then selects the table which automatically populates the schema as the output schema of the database plugin.

Case #3

• Shankar is using the Hydrator Studio instance to build a pipeline, he is building a batch pipeline for processing data from the Stream
• Albert is also using the same instance of Hydrator Studio to build his pipeline, he is building a real-time pipeline for processing data from Twitter
• Both Albert and Shankar have complex pipelines to build and they want to ensure that their work is not lost, so they are periodically saving it as draft
• When both of them save drafts asynchronously to each other, the draft from each are visible to each other. 
  

User Stories

There are hydrator specific functionalities which could leverage CDAP’s features.

• Drafts

  • User wants to add a new draft or save the pipeline he is working as a draft

  • User can update an existing draft of a pipeline as new version – previous version of pipelines are saved (upto 20 versions)
  • User can go back to previous version of draft or for any version of draft
  • User wants to retrieve the latest version of draft for a pipeline
  • User wants to view all available pipeline drafts across all users
  • User wants the ability to write a pipeline draft
  • User has access to only those pipelines that are available in the namespace the user is in.
• Plugin Output Schema
  • User using DB-Source wants to enter connection-string, table name and automatically populate table schema information. 
  • User using TeraData-Source wants to enter connection-string, table name and automatically populate table schema information. 
• List Field values 
  • User provides connection-string, user-name and password and expects list of available tables returned in DB-Source.
    
    

 Proposed REST APIs

{ 
"config": {...},
"message" : "..."
}

{ 
"config": {...},
"message" : "..."
}

[
{
"message" : "...",
"config": {
 "source" : {
    ....
  }, 
 "transforms" : [...],
 "sinks" [...]
 "connections" : [..]
 }
}, 
... 
]

{ 
"message" : "...",
"config": {
 "source" : {
    ....
  }, 
 "transforms" : [...],
 "sinks" [...]
 "connections" : [..]
 }
}

[ 
  "streamToTPFS",
  "DBToHBase",
   ...
]

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
},
"jdbcConnectionString": "...", 
 "jdbcPluginName": "...", 
 "tableName" : "..."
}

{ 
  "field1" : Integer,
  "field2" : String,
   ...
  "fieldN" : Double
}

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
}
"connectionString":   ..., 
 "username": ..., 
 "password" : ...
}

[ 
  "tableA",
  "tableB"
   ...
  "tableN" 
]

[ 
  "4as432-are425-..",

  "4az422-are425-.."

]

{ 
  "timestamp":"1455832171"

}

[ 
 purchases, 

 history, 

]

{ 
  "key" : "...",
  "value" : "..."
}

{ 
  "key" : "...",
  "value" : "..."
}