Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

This document provide a step-by-step guide of how to view Cloud Data Fusion pipeline logs in Stackdriver.

Overview

Cloud Data Fusion integrates with Stackdriver Logging, which allows you to collect and view your Cloud Data Fusion pipeline logs. Cloud Data Fusion’s Stackdriver logging contains logs from various resources such as Yarn and HDFS from the Dataproc cluster. Viewing these logs can provide insight to the lifecycle and resource usage of your pipeline on the Dataproc cluster, which can help you debug and fine-tune your pipeline.

Enabling Stackdriver Logging

To use Stackdriver Logging with your Cloud Data Fusion pipeline, you need to enable Stackdriver Logging when you create your Cloud Data Fusion instance.

  1. In the GCP Console Data Fusion Instances page, click Create instance.

  2. Click Show advanced options.

  3. Under Logging and monitoring, click Enable Stackdriver logging service.

Cloud Data Fusion Beta supports enabling/disabling Stackdriver Logging only during instance creation. After you create a Cloud Data Fusion instance, you can’t update the Stackdriver Logging option.

Viewing logs in Stackdriver

Every Cloud Data Fusion pipeline run is assigned a unique RunID. After you deploy and run your pipeline, find the RunID of the pipeline run you would like to view logs for. In Stackdriver, view the logs for the pipeline run with that RunID.

Getting your pipeline RunID

  1. After your pipeline has successfully run, click Summary.

  2. In the Summary page, click the Table link.

  3. Click on the RunId link to view and copy to clipboard.

Viewing logs

In the Stackdriver UI, in the Logs View page, select Cloud Dataproc Cluster → cdap-<pipeline-name>-<runId>.

Filtering logs

You can use the filter options filter to filter the logs you see. You can filter by component such as datafusion-pipeline-logs or yarn-resourcemanager logs, or you can filter by various log levels. Use the dropdown menu to choose a filter.

Downloading logs

You can download logs from Stackdriver. Click the Download logs option at the top.

  • No labels