This document provide provides a step-by-step guide of how to view viewing Cloud Data Fusion pipeline logs in Stackdriver.
...
To use Stackdriver Logging with your Cloud Data Fusion pipeline, you need to enable Stackdriver Logging when you create your Cloud Data Fusion instance.
...
Data Fusion
...
instance
...
Click Show advanced options.
...
.
...
| Note |
|---|
Cloud Data Fusion Beta supports enabling/disabling Stackdriver Logging only during instance creation. After you create a Cloud Data Fusion instance, you can’t update the Stackdriver Logging option. |
In the GCP Console Data Fusion Instances page, click Create instance.
Click Show advanced options.
Under Logging and monitoring, click Enable Stackdriver logging service.
...
Viewing logs in Stackdriver
...
After your pipeline has successfully run, click Summary.
In the Summary page, click the Table link.
Click on the
RunIdlink to view and copy to clipboard.
Viewing logs
...
Go to the Stackdriver
...
Logging > Logs Viewer page in the GCP Console.
Select
Cloud Dataproc Cluster
...
> cdap-<pipeline-name>-<runId>.
...
Filtering logs
You can use the filter options filter to filter the logs you see. You can filter by component such as datafusion-pipeline-logs or yarn-resourcemanager logs, or you can filter by various log levels. Use the dropdown menu to choose a filter.
...