...
To start the preview for an application:
Request Method and EndpointCode Block POST /v3/namespaces/{namespace-id}/apps/{app-id}/preview where namespace-id is the name of the namespace app-id is the name of the application for which preview is to be previewed Note that user will have to provide the app-id for the preview request. Response will contain the CDAP generated unique preview_-id which can be used further to get the preview data per stage.Request body will contain the application configuration along with following additional configs for the preview.
a. Connection configuration in the application JSON can be updated to specify the input mock data for the destination stage of the connection.
Code Block Consider pipeline represented by following connections. "connections":[ { "from":"FTP", "to":"CSVParser" }, { "from":"CSVParser", "to":"Table" } ] Now if user wants to provide the input data to the CSVParser instead of reading it from FTP source the inputData can be provided as: "connections":[ { "from":"FTP", "to":"CSVParser", "inputData" : [ {"offset": 1, "body": "100,bob"}, {"offset": 2, "body": "200,rob"}, {"offset": 3, "body": "300,tom"} ] }, { "from":"CSVParser", "to":"Table" } ] The above configuration indicates that connection from FTP to CSVParser need to be updated to read the inputData instead of reading it from FTP Source directly.
b. In case there are multiple sources in the pipeline or multiple stages reading from the same source and user wants to provide the mock data for all of them, then every connection emitted from the source need to have the inputData associated with it.Code Block Consider the following pipeline: "connections": [ { "from": "S3 Source", "to": "Log Parser" }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs" } ] Now if user want to preview the pipeline but do now want to read the data from the S3 Source, connections can be updated with the inputData information as "connections": [ { "from": "S3 Source", "to": "Log Parser", "inputData": [ "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326" ] }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs", "inputData": [ "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326" ] } ]
c. If user wants to use actual source for the preview, the number of records read from the source can be limited by the numOfRecords property.Code Block "connections": [ { "from": "S3 Source", "to": "Log Parser", "numOfRecords": 10 }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs", "numOfRecords": 10, "inputData": [ "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326" ] } ] In the above example configuration, 10 records will be read from the S3 Source in order to pass them to the Log Parser, however since the inputData is specified for the connection S3 Source to Raw Logs, inputData will be passed to Raw Logs without reading from the S3 Source.d. If user do not want to write to the sink, following are few possible approaches:
Code Block Approach a) Specify list of sinks to ignore using ignoreSinks property. "preview" : { "ignoreSinks" : ["Raw Logs"] } Approach b) For each connection to the Sink we can add the ignoreConnection property and set it to true as "connections": [ { "from": "S3 Source", "to": "Log Parser", "numOfRecords": 10 }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs", "numOfRecords": 10, "ignoreConnection": "true" } ] In the example configuration above, preview will write to the Aggregated Results, however would not write to the Raw Logs.
e. Preview single stage:Code Block Consider pipeline connections: "connections":[ { "from":"FTP", "to":"CSVParser" }, { "from":"S3", "to":"CSVParser" }, { "from":"CSVParser", "to":"Table" } ] CSVParser in the above pipeline has two input connections, one from FTP and another from S3. In order to preview the single stage CSVParser following configurations can be specified - "connections":[ { "from":"FTP", "to":"CSVParser", "inputData" : [ {"offset": 1, "body": "100,bob"}, {"offset": 2, "body": "200,rob"}, {"offset": 3, "body": "300,tom"} ] }, { "from":"S3", "to":"CSVParser", "inputData" : [ {"offset": 1, "body": "500,milo"}, {"offset": 2, "body": "600,whitney"}, {"offset": 3, "body": "700,yosemite"} ] }, { "from":"CSVParser", "to":"Table" } ] "preview": { "endStages": ["CSVParser"] } Note that in 3.5, only one stage can be provided as a endStages and when endStages is specified, inputData must be provided for all the incoming connections to that stage.
f. NOT in 3.5 In order to execute the section of the pipeline, endStages can be provided.Code Block Consider the following pipeline: "connections": [ { "from": "S3 Source", "to": "Log Parser" }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Python Evaluator" }, { "from": "GroupPython By AggregatorEvaluator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs" } ] (S3 Source) --------->(Log Parser)--------->(Group By Aggregator)--------->(Python Evaluator)--------->(Aggregated Result) | | --------->(Raw Data) Now if user wants to preview (Log Parser)--------->(Group By Aggregator) section of the pipeline, endStages can be provided as Group By Aggregator using following configurations: "connections": [ { "from": "S3 Source", "to": "Log Parser", "inputData": [ "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0800] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - bob [10/Oct/2000:14:55:36 -0710] GET /apache_pb.gif HTTP/1.0 200 2326", "127.0.0.1 - tom [10/Oct/2000:23:55:36 -0920] GET /apache_pb.gif HTTP/1.0 200 2326" ] }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3Python SourceEvaluator", "to": "Raw Aggregated Result" }, { "from": "S3 Source", "to": "Raw Logs" } ] "preview": { "endStages": ["Group By Aggregator"], "ignoreSinks": ["Raw Logs"] }- Once the preview is started, the unique preview id will be generated for it. Preview id could be of the form: namespace_id.app_id.preview. The runtime information (<Preview_id, STATUS) for the preview will be generated and will be stored (in-memory or disk). While preview is running if user again sends the preview request for the same application and the STATUS is RUNNING, user will get 403 status code with "Preview already running for application." error message. This enforces only one preview is running at any instance of time for a given application in a given namespace.
If the startStage in the preview configurations is not SOURCE plugin, then the preview system will generate the MOCK source in the pipeline which will read the JSON records specified in the inputData field and convert them into the StructureRecord.However consider the modified version of the above pipeline: "connections": [ { "from": "S3 Source", "to": "Log Parser" }, { "from": "Log Parser", "to": "Group By Aggregator" }, { "from": "Group By Aggregator", "to": "Aggregated Result" }, { "from": "S3 Source", "to": "Javascript Transform" }, { "from": "Javascript Transform", "to": "Raw Logs" } ] (S3 Source) --------->(Log Parser)--------->(Group By Aggregator)--------->(Python Evaluator)--------->(Aggregated Result) | | --------->(Javascript Transform)---------->(Raw Logs) In this case user still wants to execute the section (Log Parser)--------->(Group By Aggregator) of the pipeline. OPEN QUESTION: How to avoid execution of branch (S3 Source)--------->(Javascript Transform)---------->(Raw Logs) One way is to add ignoreConnection property for the (S3 source)--------->(Javascript Transform) connection. - Once the preview is started, the unique preview-id will be generated for it. The runtime information (<preview-id, STATUS) for the preview will be generated and will be stored (in-memory or disk).
- Once the preview execution is complete, its runtime information will be updated with the status of the preview (COMPLETED or FAILED).
To get the status of the preview
Request Method and EndpointCode Block GET /v3/namespaces/{namespace-id}/apps/{app-id}/previews/{preview-id}/status where namespace-id is the name of the namespace app-id is the name of the application for which preview data is to be requested preview-id is the id of the preview for which status is to be requestedResponse body will contain JSON encoded preview status and optional message if the preview failed.
Code Block 1. If preview is RUNNING { "status": "RUNNING" } 2. If preview is COMPLETED { "status": "COMPLETED" } 3. If preview FAILED { "status": "FAILED" "errorMessage": "Preview failure root cause message." }To get the preview data for stage:
Request Method and EndpointCode Block GET /v3/namespaces/{namespace-id}/apps/{app-id}/previews/{preview-id}/stages/{stage-name} where namespace-id is the name of the namespace app-id is the name of the application for which preview data is to be requested stage-name is the unique name used to identify the stageResponse body will contain JSON encoded input data and output data for the stage as well as input and output schema.
Code Block { "inputData": [ {"first_name": "rob", "zipcode": 95131}, {"first_name": "bob", "zipcode": 95054}, {"first_name": "tom", "zipcode": 94306} ], "outputData":[ {"name": "rob", "zipcode": 95131, "age": 21}, {"name": "bob", "zipcode": 95054, "age": 22}, {"name": "tom", "zipcode": 94306, "age": 23} ], "inputSchema": { "type":"record", "name":"etlSchemaBody", "fields":[ {"name":"first_name", "type":"string"}, {"name":"zipcode", "type":"int"} ] }, "outputSchema": { "type":"record", "name":"etlSchemaBody", "fields":[ {"name":"name", "type":"string"}, {"name":"zipcode", "type":"int"}, {"name":"age", "type":"int"} ] } }To get the logs/metrics for the preview:
Request Method and EndpointCode Block GET /v3/namespaces/{namespace-id}/apps/{app-id}/preview/logs GET /v3/namespaces/{namespace-id}/apps/{app-id}/preview/metric where namespace-id is the name of the namespace app-id is the name of the application for which preview data is to be requestedResponse would be similar to the regular app.
...