Running pipelines in a custom network

This page describes how to configure a pipeline to run on a Dataproc cluster in a specific network. This can be done by setting the network using runtime arguments, or creating a custom profile.

Setting network for a single pipeline

1. In the Cloud Data Fusion UI, navigate to the detail page for the pipeline you would like to run.

2. In the Configure menu, click on Compute config

3. Click on Customize. Under General Settings, enter the desired value for the network.

   

Creating a custom profile

1. In the Cloud Data Fusion UI, navigate to the System Admin page using the top nav bar.

2. Click on the Configuration tab, then click on Create New Profile in the System Compute Profiles section.

   

3. Click on Google Cloud Dataproc.

4. Enter a Profile label and a Description.

5. Enter the Network that you would like your pipelines to run in.

   

6. Click on the Create at the bottom of the page. The other settings will be automatically detected.

7. Navigate to the pipeline detail page for the pipeline you would like to run.

8. Click on the Configure button, select the profile you just created, then save.