This article will outline the steps required to connect to sources and sinks that reside outside of the customer project. The workflow within Data Fusion is the same, you . You will simply need to provide access to a couple of service accounts so CDF can access them. Note that Data Fusion by default has access to read and write to Big Query/GCS/Pub-Sub/Spanner/BigTable on the project where the Data Fusion instance is created, so these steps are not needed for sources and sinks in the customer project.
Service Account Description
There is some confusion regarding the various service accounts Data Fusion creates/uses to operate. This table provides a breakdown for each service account and its function:
Service account name format | Description | Uses in CDF |
|---|---|---|
service-<project_number>@gcp-sa-datafusion.iam.gserviceaccount.com | Service account used by CDF in the tenant project to access resources in the customer project. | This account is used to:
|
<project_number>-compute@developer.gserviceaccount.com | Default service account used by the Dataproc VMs. | This account is used to:
|
...
Navigate to the customer project that contains the CDF instance and copy the project number (this is found on the Home Page in the Project Info card).
Navigate to the project that contains the resources you would like to interact with.
In the sidebar, click on ‘IAM & Admin’ .
Click on ‘Add’ at the top of the page.
Provide the first service account name from the table above, be . Be sure to replace <project_number> with the actual number you obtained in step 1.
Grant the
Adminrole for the resource you would like to interact with. Ex.BigQuery Adminfor reading/writing to BigQuery. For BigQuery, you will also need to grant theBigQuery Data Ownerrole as well.Repeat steps 5 & 6 for the second service account in the table above.
In your data pipeline, ensure you define the correct Project Id for the sources/sinks. Using ‘auto-detect’ will default to the customer project that contains the CDF instance.
...