Insufficient 'DISKS_TOTAL_GB' quota when running a pipeline

Problem

Pipelines fail with the following error in the log:

This error means that the Dataproc cluster provisioned by your pipeline would cause it to exceed the GCE quota for compute disks. Since the Dataproc cluster cannot be provisioned, the pipeline fails.

Solution(s)

There are two ways to resolve this issue: raise your project quota or configure Dataproc disk sizes.

Raise your project quota

The quota that must be raised for this error is Persistent disk standard (GB). There are both project wide and regional quotas. You can see that documentation here for more information as well as steps on how to raise it: https://cloud.google.com/compute/quotas.

Configure Dataproc disk sizes

The size of the Dataproc cluster can be configured through the use of cluster properties in order to keep it under quota. The defaults can be overridden by adding runtime arguments to the pipeline as described in Setting custom Dataproc cluster properties.

In this case, the relevant properties are:

system.profile.properties.masterDiskGB

system.profile.properties.workerDiskGB

Set these to a low enough value (in Gb) so that the resultant Dataproc cluster remains under quota.