Background

BigQuery imposes a limit of 10,000 paths in the source URIs when writing to it. For CDF this means that there cannot be more than 10,000 splits/partitions going into a BQ sink. The number of partitions in a pipeline can be determined by various things such as:

1. Source configuration

2. Joins

3. Aggregations

This article will cover solutions for the first point, source configuration, in order to avoid running into this issue.

Problem

We will be using the GCS Source as an example but the same theory applies to other batch sources. There are two situations that can lead to the GCS source generating more than more than 10,000 partitions:

1. There are over 10,000 small input files which are each being treated as a their own partition, resulting in more than 10,000 partitions.

2. There are less some large input files which are getting into split more than 10,000 partitions. By default, GCS Source has a max split size of 128 MB, so a 1 GB file will generate roughly 8 partitions.

Solution(s)

The solution depends on which situation it is. If the customer is unsure, then they can implement both solutions:

• If there are many small files, then we need to set the minimum split size property in the filesystem. This will combine many smaller files into one partition which results in less overall partitions. This is accomplished by setting the value of File System Properties in the Advanced section of the GCS source to {"mapreduce.input.fileinputformat.split.minsize": 134217728 }. The value '134217728' is in bytes which translates to 128 MB.

• If there are large files, then we need to increase the Maximum Split Size. This will decrease the number of splits generated from a large file. This is accomplished by setting the value of Maximum Split Size in the Advanced section of the GCS source to a value greater than 134217728 bytes (default is 128 MB).