Running pipelines in a custom network

This page describes how to configure a pipeline to run on a Dataproc cluster in a specific network. This can be done by setting the network using runtime arguments or by creating a custom profile.

Setting network for a single pipeline

1. In the Pipeline Studio, navigate to the detail page for the pipeline you would like to run.

2. In the Configure menu, click on Compute config.

3. Click on Customize. Under General Settings, enter the desired value for the network.

   Open

   

Creating a custom profile

1. In the Pipeline Studio, navigate to the System Admin page using the top nav bar.

2. Click on the Configuration tab, and then click on Create New Profile in the System Compute Profiles section.

   Open

   

3. Click on Google Cloud Dataproc.

4. Enter a Profile label and a Description.

5. Enter the Network that you would like your pipelines to run in.

    

6. Click on the Create at the bottom of the page. The other settings will be automatically detected.

7. Navigate to the pipeline detail page for the pipeline you would like to run.

8. Click on the Configure button, select the profile you just created, then save.