Page Comparison

...

endStages(Optional): List of stages(inclusive) till which preview need to be executed. If not given, preview will be executed till the sink.
realLookups: List of datasets to be used from the real space for the lookup purpose only. This is required field, since if it is not given the datasets from the preview space will be used which will be empty.
realExternalWrites(Optional): List of EXTERNAL datasets to which the actual writes need to happen. If user does not want to write to the external dataset, we can replace the sink with the /dev/null sink in the pipeline configurations. Note that this is only required for the external datasets. The CDAP internal datasets will be created in the preview space only, so even if we write to them, those writes will not be visible in the real space.
numRecords(Optional): Number of records to be read from the source for the preview. If not given we will read ALL the records pre-configured number of records(say 100) from the source.
outputs(Optional): User can provide test data to preview using outputs configurations.

...

Preview entire pipeline. Read from File source and write to the database.

Code Block

language	java

a. 
"appPreviewConfig": {
   "realLookups": ["File"],
   "realExternalWrites": ["DBSink"]
}
 
b. Same can be achieved by
"appPreviewConfig": {
   "endStages": ["DBSink"],
   "realLookups": ["FTP"],
   "realExternalWrites": ["DBSink"]
}

Preview pipeline with test records (in JSON format) instead of reading it from File.

Code Block

language	java

"appPreviewConfig": {
   "endStages": ["DBSink"],
   "realLookups": ["File"],
   "realExternalWrites": ["DBSink"],
   "outputs": {
      "File": {
         "data": [
            {"offset": 1, "body": "100,bob"},
            {"offset": 2, "body": "200,rob"},
            {"offset": 3, "body": "300,tom"}
         ]
      }
   }
}
 
Above pipeline uses test records provided in the outputs section instead of reading from the actual Files.

Preview pipeline without writing to the Database.

Code Block

language	java

"appPreviewConfig": {
   "endStages": ["DBSink"],
   "realLookups": ["File"],
   "outputs": {
      "File": {
         "data": {[
            {"offset": 1, "body": "100,bob"},
            {"offset": 2, "body": "200,rob"},
            {"offset": 3, "body": "300,tom"}
         }]
      }
   }
}
 
Since the realExternalWrites is not provided, app will be reconfigured to write to /dev/null

Preview single stage CSVParser

Code Block

language	java

"appPreviewConfig": {
   "endStages": ["CSVParser"], // Note that endStages is updated to CSVParser to just preview the single stage.
   "realLookups": ["File"],
   "outputs": {
      "File": {
         "data": [
            {"offset": 1, "body": "100,bob"},
            {"offset": 2, "body": "200,rob"},
            {"offset": 3, "body": "300,tom"}
         ]
      }
   }
}

...

Versions Compared

Old Version 3

New Version Current

Key