...
Orchestrating a Data Fusion pipeline from external services is a common use case. For example, Composer might start data pipelines based on data availability in GCS bucket and then wait for the pipeline to complete. The guide This article walks you through the APIs needed to trigger the pipeline and wait for it to complete, monitor the pipeline run, and get the pipeline status when it completes.
Approach
Triggering of the pipeline should get the Run ID of the pipeline that is started. To get the Run ID, use the following endpoint:
| Code Block |
|---|
POST v3/namespaces/<namespaceid>/start Request Body: ['{“appId”:”<pipelinename>”, “programType”:”workflow”, “programId”:”DataPipelineWorkflow”, "runtimeargs": { "arg1":"val1" } }'] |
Sample Call
| Code Block |
|---|
Request: curl -X POST http://localhost:11015/v3/namespaces/default/start
-d '[{"appId": "Test", "programType": "workflow", "programId":"DataPipelineWorkflow", "runtimeargs": { "path":"gs://mybucket/csv/data.txt" }}]'
Response: [{"statusCode":200,"runId":"e6523380-a73a-11ea-a21e-acde48001122","appId":"Test","programType":"Workflow","programId":"DataPipelineWorkflow"}] |
...