Versions Compared

Old Version 35

changes.mady.by.user Sagar Kapare

Saved on

compared with

New Version Current

changes.mady.by.user Sagar Kapare

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. To start the preview for an application: 
    Request Method and Endpoint 

    Code Block
    POST /v3/namespaces/{namespace-id}/apps/{app-id}/preview
where namespace-id is the name of the namespace
      app-id is the name of the application for which preview is to be previewed

Note that user will have to provide the app-id
for the preview request.
 
Response will contain the CDAP generated unique preview-id which can be used further to get the preview data per stage.

    Request body will contain the application configuration along with following few additional configs for the preview section.

    a.  Connection configuration in the application JSON can be updated to specify the input mock data for the destination stage of the connection.

    Code BlockConsider pipeline represented by following connections. "connections":[

    Code Block
    {
    "artifact":{ 
      "name":"cdap-data-pipeline",
      "version":"3.5.0-SNAPSHOT",
    {  "scope":"SYSTEM"
       },
    "fromname":"FTPMyPipeline",   
         "toconfig":"CSVParser"{
        "connections":[ },
         { 
             "from":"CSVParserFTP",
            "to":"TableCSVParser"
         },
      	]  Now if{ user
wants to provide the input data to the CSVParser instead of reading it from FTP source the inputData can be provided as:
"connections":[ "from":"CSVParser",
            "to":"Table"
         }
 {       ],
        "from":"FTP",stages":[ 
         { 
            "toname":"CSVParserFTP",
			"inputData"
        : [   "plugin":{ 
               "name":"FTP",
             {"offset": 1, "bodytype": "100,bobbatchsource"},
               "label":"FTP",
                  {"offset"artifact":{ 
2, "body": "200,rob"},                "name":"core-plugins",
                  {"offsetversion": 3, "body": "300,tom"}"1.4.0-SNAPSHOT",
                  "scope":"SYSTEM"
           ]    },
     },          "properties":{ 
                  "fromreferenceName":"CSVParsermyfile",
                  "topath":"Table/tmp/myfile"
         }      }
	]   The above configuration indicates that connection from FTP to CSVParser},
need to be updated to read the inputData instead of reading it from FTP Source directly.
    b. In case there are multiple sources in the pipeline or multiple stages reading from the same source and user wants to provide the mock data for all of them, then every connection emitted from the source need to have the inputData associated with it.
    Code Block
    Consider the following pipeline:
 "connections": [
        {
            "from": "S3 Source",
            "to": "Log Parser" "outputSchema":"{\"fields\":[{\"name\":\"offset\",\"type\":\"long\"},{\"name\":\"body\",\"type\":\"string\"}]}"
         },
         { 
            "fromname": "Log ParserMyCSVParser",
            "toplugin":{ "Group
By Aggregator"         },     "name":"CSVParser",
   {             "fromtype": "transform"Group,
By Aggregator",              "tolabel": "Aggregated Result""CSVParser",
             },  "artifact":{ 
     {             "fromname": "S3 Source"transform-plugins",
                  "toversion": "Raw Logs""1.4.0-SNAPSHOT",
        }     ]  Now if user want to preview the pipeline but do now want to read the data from the S3 Source, connections can be updated with the inputData information as 
 "connections": [ "scope":"SYSTEM"
               },
               "properties":{ 
        {             "fromformat": "S3 SourceDEFAULT",
                  "to": "Log Parser",
			"inputData": [
							"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"
						 ]	"schema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
                  "field":"body"
               }
   },         {},
            "fromoutputSchema":"{\"type\": "Log Parser",
            "to": "Group By Aggregator"
        },
        {\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}"
         },
       "from": "Group By{ Aggregator",
            "toname": "Aggregated ResultMyTable"
,
       },     "plugin":{ 
  {             "fromname": "S3 SourceTable",
               "totype": "Raw Logsbatchsink",
			"inputData": [
							"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"	
						 ]               "label":"Table",
               "artifact":{ 
                  "name":"core-plugins",
             }     ]
    c. If user wants to use actual source for the preview, the number of records read from the source can be limited by the numOfRecords property. 
    Code Block
     "connections": [
  "version":"1.4.0-SNAPSHOT",
     {             "fromscope": "S3 Source",
         SYSTEM"
  "to": "Log Parser", 			"numOfRecords": 10         },
        {        "properties":{ 
   "from": "Log Parser",             "toschema": "Group By Aggregator"
        },
  "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
     {             "fromname": "Group By Aggregatormytable",
            "to": "Aggregated Result"         },"schema.row.field":"id"
             {  }
          "from": "S3 Source"},
            "to"outputSchema":"{\"type\": "Raw Logs",
            "numOfRecords": 10,
			"inputData": [
							"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"	

						 ]
  \"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
            "inputSchema":[ 
               { 
    }     ]   In the above example configuration, 10 records will be read from the S3 Source in order to pass them to the Log Parser, however since the inputData is specified for the connection S3 Source to Raw Logs, inputData will be passed to Raw Logs without reading from the S3 Source.

    d. If user do not want to write to the sink, following are few possible approaches:

    Code Block
    Approach a) Specify list of sinks to ignore using ignoreSinks property.

"preview" : {
				"ignoreSinks" : ["Raw Logs"]  
			}

Approach b) For each connection to the Sink we can add the ignoreConnection property and set it to true as
 "connections": [
   "name":"id",
                  "type":"int",
                  "nullable":false
               },
               { 
                  "name":"name",
     {             "fromtype": "S3 Sourcestring",
              "to": "Log Parser",
			"numOfRecords": 10    "nullable":false
        },       }
 {           ]
 "from": "Log Parser",      }
      "to": "Group By Aggregator"],
       "preview": {
},         {      "startStages":    ["MyCSVParser"],
			   "fromendStages": "Group By Aggregator",["MyTable"],
               "touseSinks": ["Aggregated Result"MyTable"],
         },      "outputs": {
								"FTP": {
										"data":	[
            "from": "S3 Source",                  "to": "Raw Logs", 			{"numOfRecordsoffset": 101, 			"ignoreConnectionbody": "true100,bob"},
        }     ]  In the example configuration above, preview will write to the Aggregated Results, however would not write to the Raw Logs.
    e. Preview single stage:
    Code Block
    Consider pipeline connections:
"connections":[ 			{"offset": 2, "body": "200,rob"},
           {               "from":"FTP",             "to":"CSVParser"			{"offset": 3, "body": "300,tom"}
         },          {               "from":"S3",
            "to":"CSVParser"
         },
         {  
            "from":"CSVParser",
            "to":"Table"
         }
      	]
 
CSVParser in the above pipeline has two input connections, one from FTP and another from S3. 
In order to preview the single stage CSVParser following configurations can be specified -
 
"connections":[ 			],
										"schema": {
													"type" : "record",
													"fields": [
																{"name":"offset","type":"long"},
															  	{"name":"body","type":"string"}
															 ]
												  }
								}
						}	
			}
 
    }
}

    a.  Simple pipeline

    Code Block
    Consider simple pipeline represented by following connections.

(FTP)-------->(CSV Parser)-------->(Table)
 
CASE 1: To preview the entire pipeline:
 
"preview": {
          {     "startStages": ["FTP"],
			        "fromendStages": ["FTPTable"],
               "touseSinks": ["CSVParserTable"],
			   "inputDatanumRecords": 10,
:			 [  "programName": "SmartWorkflow",  // The program to be previewed
			   "programType": "workflow"		// The program type to be previewed	
			}
 
CASE 2: To preview section of the pipeline: (CSV  {"offset": 1, "body": "100,bob"},Parser)-------->(Table)
 
"preview": {
               "startStages": ["CSVParser"],
			   "endStages": ["Table"],
              { "offsetuseSinks": 2, "body": "200,rob"}, ["Table"],
               "outputs": {
								"FTP": {
										"data":	[
                                 			{"offset": 31, "body": "300100,tombob"},
                              ]   			{"offset":       },
         {  
            "from":"S3",
            "to":"CSVParser",
			"inputData"  : [2, "body": "200,rob"},
                                     			{"offset": 13, "body": "500300,milotom"},
                                  {"offset			],
										"schema": 2, "body"{
													"type" : "600,whitneyrecord"},
													"fields":                                  {"offset": 3, "body": "700,yosemite"}
                              ][
																{"name":"offset","type":"long"},
															  	{"name":"body","type":"string"}
															 ]
												  }
								}
						}	
			}
 
CASE 3: To preview only single stage (CSV Parser) in the pipeline:
 
"preview": {
          },     "startStages": ["CSV Parser"],
			   "endStages": ["CSV Parser"],
  {             "outputs": {
"from":"CSVParser",								"FTP": {
										"data":	[
                "to":"Table"          }       			]
{"previewoffset": {1, 	"endStagesbody": ["CSVParser100,bob"]
},
  Note that in 3.5, only one stage can be provided as a endStages and when endStages is specified, inputData must be provided for all the incoming connections to that stage.
    f. NOT in 3.5   In order to execute the section of the pipeline, endStages can be provided.
    Code Block
    Consider the following pipeline 			{"offset": 2, "connectionsbody": ["200,rob"},
        {             "from": "S3 Source",           			{"offset": 3, "tobody": "Log Parser"300,tom"}
        },         {             			],
"from										"schema": "Log Parser",
            "to": "Group By Aggregator"
        },
		{
            "from": "Group By Aggregator",
            "to": "Python Evaluator"
        },
        {
            "from": "Python Evaluator",
            "to": "Aggregated Result"
        },
        {
            "from": "S3 Source",{
													"type" : "record",
													"fields": [
																{"name":"offset","type":"long"},
															  	{"name":"body","type":"string"}
															 ]
												  }
								}
						}	
			}
 
CASE 4: To verify if records are read correctly from FTP:
"preview": {
               "tostartStages": ["Raw LogsFTP"],
			     "endStages": ["FTP"],
			   "numOfRecords": 10
			}
 
CASE 5: To verify ]the  data  is (S3getting Source) --------->(Log Parser)--------->(Group By Aggregator)--------->(Python Evaluator)--------->(Aggregated Result)written to Table properly:
"preview": {
             | 			|
			--------->(Raw Data)	
 
Now if user wants to preview (Log Parser)--------->(Group By Aggregator) section of the pipeline, endStages can be provided as Group By Aggregator using following configurations:
 
"connections": [
 "startStages": ["Table"],
			   "endStages": ["Table"],
               "useSinks": ["Table"],
       {             "fromoutputs": "S3 Source",{
								"CSV Parser": {
										"data":	[
                  "to": "Log Parser", 			"inputData": [ 							"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326", 							"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
							"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"
						 ]	{"id": 1, "name": "bob"},
                            },         {
            "from			{"id": 2, "name": "Log Parserrob"},
            "to": "Group By Aggregator"         },         {
            "from": "Group By Aggregator",
            "to			{"id": 3, "name": "Aggregated Resulttom"}
        },         {             "from": "Python Evaluator",
            "to": "Aggregated Result"
        },
        {
            "from": "S3 Source",
            "to": "Raw Logs"
        }
    ]

"preview": {
	"endStages": ["Group By Aggregator"],
    "ignoreSinks": ["Raw Logs"]    
}
 
  
However consider the modified version of the above pipeline:
 
 "connections": [
        {			],
										"schema": {
													"type" : "record",
													"fields": [
																{"name":"id","type":"long"},
															  	{"name":"name","type":"string"}
															 ]
												  }
								}
						}	
			}

    b. Fork in the pipeline (multiple sinks)

    Code Block
    Consider the following pipeline:
 
(S3 Source) --------->(Log Parser)--------->(Group By Aggregator)--------->(Python Evaluator)--------->(Aggregated Result)
            |
			|
			--------->(Javascript  "from": "S3 Source",
            "to": "Log Parser"Transform)--------->(Raw Data)
 
CASE 1: To preview entire pipeline
"preview": {
         },      "startStages": ["S3  {Source"],
			   "endStages":          "from": "Log Parser",["Aggregated Result", "Raw Data"],
               "touseSinks": ["Group By Aggregator"
        },
        {
            "from": "Group By Aggregator",Aggregated Result", "Raw Data"], // useSinks seem redundant as endStages is there which can control till what point the pipeline need to run
			   "numOfRecords": 10
			}
 
CASE 2: To mock the source
"preview": {
               "tostartStages": ["AggregatedLog ResultParser", "Javascript Transform"],
			   "endStages":   }["Aggregated Result",      "Raw Data"],
  {             "fromuseSinks": ["S3Aggregated SourceResult",
            "to":Raw "Javascript Transform"
        },
     Data"],
  {             "fromoutputs": "Javascript Transform",
            "to": "Raw Logs"
        }
    ]
 
 
(S3 Source) --------->(Log Parser)--------->(Group By Aggregator)--------->(Python Evaluator)--------->(Aggregated Result)
 {
								"S3 Source": {
										"data":	[
												"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326",
												"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
												"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"
          |
			| 			--------->(Javascript Transform)---------->(Raw Logs)   In this case user still wants to execute the section (Log Parser)--------->(Group By Aggregator) of the pipeline.
OPEN QUESTION: How to avoid execution of branch (S3 Source)--------->(Javascript Transform)---------->(Raw Logs)
One way is to add ignoreConnection property for the (S3 source 			],
										"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"log_line","type":"string"}
															 ]
												  }
								}
						}	
			}
 
 
CASE 3: To preview the section of the pipeline (Log Parser)--------->(JavascriptGroup By Transform) connection.
    Once the preview is started, the unique preview-id will be generated for it. The runtime information (<preview-id, STATUS) for the preview will be generated and will be stored (in-memory or disk). 
    Once the preview execution is complete, its runtime information will be updated with the status of the preview (COMPLETED or FAILED).

  2. To get the status of the preview
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/apps/{app-id}/previews/{preview-id}/status
where namespace-id is the name of the namespace
      app-id is the name of the application for which preview data is to be requested
	  preview-id is the id of the preview for which status is to be requested

    Response body will contain JSON encoded preview status and optional message if the preview failed.

    Code Block
    1. If preview is RUNNING
{
	"status": "RUNNING" 
}

2. If preview is COMPLETED
{
	"status": "COMPLETED" 
}

3. If preview FAILED
{
	"status": "FAILED"
    "errorMessage": "Preview failure root cause message." 
}

  3. To get the preview data for stage:
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/apps/{app-id}/previews/{preview-id}/stages/{stage-name}
where namespace-id is the name of the namespace
      app-id is the name of the application for which preview data is to be requested
      stage-name is the unique name used to identify the stage

    Response body will contain JSON encoded input data and output data for the stage as well as input and output schema.

    Code Block
    {
	"inputData": [
    Aggregator)--------->(Python Evaluator)
"preview": {
               "startStages": ["Log Parser"],
			   "endStages": ["Aggregated Result"],
               "useSinks": ["Aggregated Result"],
               "outputs": {
								"S3 Source": {
										"data":	[
												"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326",
												"127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326",
												"127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326"
                              			],
										"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"log_line","type":"string"}
															 ]
												  }
								}
						}	
			}

CASE 4: To preview the single stage Python Evaluator
"preview": {
               "startStages": ["Python Evaluator"],
			   "endStages": ["Python Evaluator"],
               "outputs": {
								"Group By Aggregator": {
									"data": [
												{"ip":"127.0.0.1", "counts":3},
												{"ip":"127.0.0.2", "counts":4},
												{"ip":"127.0.0.3", "counts":5},
												{"ip":"127.0.0.4", "counts":6},
						 					],
									"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"ip","type":"string"},
															  	{"name":"counts","type":"long"}
															 ]
									}
								}
						}	
			}

    c. Join in the pipeline (multiple sources)

    Code Block
    Consider the following pipeline:
(Database)--------->(Python Evaluator)--------->
           										| 
												|------------>(Join)-------->(Projection)------->(HBase Sink)
												|	
           (FTP)--------->(CSV Parser)--------->
 
 
CASE 1: To preview entire pipeline
 
"preview": {
               "startStages": ["Database", "FTP"],
			   "endStages": ["HBase Sink"],
			   "useSinks": ["HBase Sink"],	
			   "numOfRecords": 10
			}


CASE 2: To mock both sources
"preview": {
               "startStages": ["Python Evaluator", "CSV Parser"],
			   "endStages": ["HBase Sink"],
			   "useSinks": ["HBase Sink"],	
               "outputs": {
								"Database": {
									"data": [
												{"name":"tom", "counts":3},
												{"name":"bob", "counts":4},
												{"name":"rob", "counts":5},
												{"name":"milo", "counts":6}
						 					],
									"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"name","type":"string"},
															  	{"name":"counts","type":"long"}
															 ]
									}
								},
 
								"FTP": {
									"data": [
												{"offset":1, "body":"tom,100"},
												{"offset":2, "body":"bob,200"},
												{"offset":3, "body":"rob,300"},
												{"offset":4, "body":"milo,400"}
						 					],
									"schema": {
													"fields": [
															  	{"name":"name","type":"string"},
															  	{"name":"offset","type":"long"}
															 ]
									}
								}
						}	
			}
 
 
CASE 3: To preview JOIN transform only
"preview": {
               "startStages": ["JOIN"],
			   "endStages": ["JOIN"],
               "outputs": {
								"Python Evaluator": {
									"data": [
												{"name":"tom", "counts":3},
												{"name":"bob", "counts":4},
												{"name":"rob", "counts":5},
												{"name":"milo", "counts":6}
						 					],
									"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"name","type":"string"},
															  	{"name":"counts","type":"long"}
															 ]
									}
								},

								"CSV Parser": {
									"data": [
												{"offset":1, "body":"tom,100"},
												{"offset":2, "body":"bob,200"},
												{"offset":3, "body":"rob,300"},
												{"offset":4, "body":"milo,400"}
						 					],
									"schema": {
													"fields": [
															  	{"name":"name","type":"string"},
															  	{"name":"offset","type":"long"}
															 ]
									}
								}
						}	
			}

    d. Preview for a single stage (TBD)

    Code Block
    Consider the pipeline containing only one transform which has no connections yet- 
 
(Javascript Transform)

The preview configurations can be provided as
 
"preview": {
               "startStages": ["Javascript Transform"],
			   "endStages": ["Javascript Transform"],
               "outputs": {
								"MockSource": {

									"data": [
												{"name":"tom", "id":3},
												{"name":"bob", "id":4},
												{"name":"rob", "id":5},
												{"name":"milo", "id":6}
						 					],
									"schema": {
													"type" : "record",
													"fields": [
															  	{"name":"name","type":"string"},
															  	{"name":"id","type":"long"}
															 ]
									}
								}
						}	
			}
  4. How to specify the input data: User can specify the input data for preview by inserting the data directly in table format in UI or can upload a file containing the records.
    When the data is inserted in Table format, UI will convert the data into appropriate JSON records.
    When user decides to upload a file, he can upload the JSON file conforming to the schema of the next stage. Ideally we should allow uploading the CSV file as well, however how to interpret the data will be plugin dependent. For example consider the list of CSV records. Now for CSVParser plugin, the entire record will be treated as body however for Table sink, we will have to split the record based on comma to create multiple fields as specified by the next stage's input schema.

  5. Once the preview is started, the unique preview-id will be generated for it. The runtime information (<preview-id, STATUS) for the preview will be generated and will be stored (in-memory or disk). 

  6. Once the preview execution is complete, its runtime information will be updated with the status of the preview (COMPLETED or FAILED).

  7. To get the status of the preview
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status
where namespace-id is the name of the namespace
	  preview-id is the id of the preview for which status is to be requested

    Response body will contain JSON encoded preview status and optional message if the preview failed.

    Code Block
    1. If preview is RUNNING
{
	"status": "RUNNING" 
}

2. If preview is COMPLETED
{
	"status": "COMPLETED" 
}

3. If preview application deployment FAILED
{
	"status": "DEPLOY_FAILED",
	"failureMessage": "Exception message explaining the failure"
}
 
4. If preview application FAILED during execution of the stages
{
	"status": "RUNTIME_FAILED",
    "failureMessage": "Failure message"
    /*
    "stages": {
		[
			"stage_1": {
				"numOfInputRecords": 10,
				"numOfOutputRecords": 10
			},
			"stage_2": {
				"numOfInputRecords": 10,
				"numOfOutputRecords": 7
			},
			"stage_3": {
				"numOfInputRecords": 7,
				"numOfOutputRecords": 4,
				"errorMessage": "Failure reason for the stage"
			}
		]
	} */
}

  8. To get the preview data for stage:
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/previews/{preview-id}/stages/{stage-id}
where namespace-id is the name of the namespace
	  preview-id is the id of the preview for which data is to be requested
      stage-id is the unique name used to identify the emitter

    Response body will contain JSON encoded input data and output data for the emitter as well as input and output schema.

    Code Block
    {
	"inputData": [
                 	{"first_name": "rob", "zipcode": 95131},
                    {"first_name": "bob", "zipcode": 95054},
                    {"first_name": "tom", "zipcode": 94306}
                 ],
	"outputData":[
					{"name": "rob", "zipcode": 95131, "age": 21},
                    {"name": "bob", "zipcode": 95054, "age": 22},
                    {"name": "tom", "zipcode": 94306, "age": 23}
				 ],
	"inputSchema": {
						"type":"record",
						"name":"etlSchemaBody",
						"fields":[
									{"name":"first_name", "type":"string"},
									{"name":"zipcode", "type":"int"}
								 ]
					},
	"outputSchema": {
						"type":"record",
						"name":"etlSchemaBody",
						"fields":[
									{"name":"name", "type":"string"},
									{"name":"zipcode", "type":"int"},
									{"name":"age", "type":"int"}
								 ]
					},
    "errorRecordSchema": {
						"type":"record",
						"name":"schemaBody",
						"fields":[
									{"name":"errCode", "type":"int"},
									{"name":"errMsg", "type":"String"},
									{"name":"invalidRecord", "type":"String"}
								 ]
					},
	"errorRecords": [  
      {  
         "errCode":12,
         "errMsg":"Invalid record",
         "invalidRecord":"{\"offset\":3,\"body\":\"This error record is not comma separated!\"}"
      }
   ],	
 
    // Logs per stage may not make sense will need to think more about it.
	"logs" : {
		"stage level logs"
	}
}

  9. To get the logs/metrics for the preview:
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs
where namespace-id is the name of the namespace
	  preview-id is the id of the preview for which data is to be requested
logs end point return the entire preview logs.
Sample response for the logs endpoint. Note that it is similar to the regular app.
[  
   {  
      "log":"This is sample log - 0",
      "offset":"0.1466470037680"
   },
   {  
      "log":"This is sample log - 1",
      "offset":"1.1466470037680"
   },
   {  
      "log":"This is sample log - 2",
      "offset":"2.1466470037680"
   },
   {  
      "log":"This is sample log - 3",
      "offset":"3.1466470037680"
   },
   {  
      "log":"This is sample log - 4",
      "offset":"4.1466470037680"
   }
]
 
GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics
where namespace-id is the name of the namespace
	  preview-id is the id of the preview for which data is to be requested
 
Sample response for the metrics. Note that it is similar to the regular app.
{
    "endTime": 1466469538,
    "resolution": "2147483647s",
    "series": [
        {
            "data": [
                {
                    "time": 0,
                    "value": 4
                }
            ],
            "grouping": {},
            "metricName": "user.Projection.records.out"
        },
        {
            "data": [
                {
                    "time": 0,
                    "value": 4
                }
            ],
            "grouping": {},
            "metricName": "user.Projection.records.in"
        },
        {
            "data": [
                {
                    "time": 0,
                    "value": 4
                }
            ],
            "grouping": {},
            	{"first_namemetricName": "rob", "zipcode": 95131},user.Stream.records.out"
        },
         {
  {"first_name": "bob", "zipcode": 95054},         "data": [
          {"first_name": "tom", "zipcode": 94306}   {
              ], 	"outputData":[ 					{"name": "rob", "zipcode": 95131, "agetime": 21}0,
                    {"namevalue": "bob", "zipcode": 95054, "age": 22}, 4
                }
           {"name": "tom", "zipcode": 94306, "age": 23}
				 ],
	"inputSchema": {
						"type":"record",
						"name":"etlSchemaBody",
						"fields":[
									{"name":"first_name", "type":"string"},
									{"name":"zipcode", "type":"int"}
								 ]
					},
	"outputSchema": {
						"type":"record",
						"name":"etlSchemaBody",
						"fields":[
									{"name":"name", "type":"string"},
									{"name":"zipcode", "type":"int"},
									{"name":"age", "type":"int"}
								 ]
					}  
}

    To get the logs/metrics for the preview:
    Request Method and Endpoint

    Code Block
    GET /v3/namespaces/{namespace-id}/apps/{app-id}/preview/logs
GET /v3/namespaces/{namespace-id}/apps/{app-id}/preview/metric
where namespace-id is the name of the namespace
      app-id is the name of the application for which preview data is to be requested ],
            "grouping": {},
            "metricName": "user.JavaScript.records.in"
        },
        {
            "data": [
                {
                    "time": 0,
                    "value": 4
                }
            ],
            "grouping": {},
            "metricName": "user.Stream.records.in"
        }
    ],
    "startTime": 0
}

    Response would be similar to the regular app.


Open Questions:

  1. How to make it easy for the user to upload the CSV file?
  2. Lookup data is the user dataset. Should we allow mocking of the look up dataset as well?