...

• Drafts

  • User wants to add a new draft or save the pipeline he is working as a draft

  • User can update an existing draft of a pipeline as new version – previous version of pipelines are saved (upto 20 versions)
  • User can go back to previous version of draft or for any version of draft
  • User wants to retrieve the latest version of draft for a pipeline
  • User wants to view all available pipeline drafts across all users
  • User wants the ability to write a pipeline draft
  • User has access to only those pipelines that are available in the namespace the user is in.
• Plugin Output Schema
  • User using DB-Source wants to enter connection-string, table name and automatically populate table schema information. 
  • User using TeraData-Source wants to enter connection-string, table name and automatically populate table schema information. 
• List Field values 
  • User provides connection-string, user-name and password and expects list of available tables returned in DB-Source.
    
  • User provides initial-address, key-space, user-name and password for key space and expects list of available column families in Cassandra.
  • User provides either kafka-broker or zoo-keeper connection string and expects the list of available kafka topics. 
  • User provides elastic search hostname and expects the list of available indexes to choose from.
  • 
    

 Proposed REST APIs

{ 
"config": {
 "source" : {
    ....
  }, 
 "transforms" : [...],
 "sinks" [...]
 "connections" : [..]
}

[ 
  "streamToTPFS",
  "DBToHBase",
   ...
]

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
},
"jdbcConnectionString": ..., 
 "jdbcPluginName": ..., 
 "tableName" : ...
}

{ 
  "field1" : Integer,
  "field2" : String,
   ...
  "fieldN" : Double
}

{ 
"artifact" : {
  "name" : "...",
  "version":"...",
  "scope":"..."
}
"connectionString":   ..., 
 "username": ..., 
 "password" : ...
}

[ 
  "tableA",
  "tableB"
   ...
  "tableN" 
]

...