...
User will need to implement a DynamicPartitioner, which is responsible for defining the PartitionKey to use for each record.
For each of the partition keys that this DynamicPartitioner returns, a partition will be created in the output PartitionedFileSet dataset.
Code Block | ||
---|---|---|
| ||
public interface DynamicPartitioner<K, V> {
PartitionKey getPartitionKey(K key, V value, long logicalStartTime);
} |
Code Block | ||
---|---|---|
| ||
public static final class CustomDynamicPartitioner implements DynamicPartitioner<byte[], Long> { @Override public PartitionKey getPartitionKey(byte[] key, Long value, long logicalStartTime) { return PartitionKey.builder().addLongField("time", logicalStartTime).addLongField("othervalue", value).build(); } } |
In the beforeSubmit() method of the user's MapReduce job, set the PartitionedFileSet dataset with the user's Dynamic Partitioner as output
...