Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

User will need to implement a DynamicPartitioner, which is responsible for defining the PartitionKey to use for each record.
For each of the partition keys that this DynamicPartitioner returns, a partition will be created in the output PartitionedFileSet dataset.

Code Block
languagejava
public interface DynamicPartitioner<K, V> {
  PartitionKey getPartitionKey(K key, V value, long logicalStartTime);
}

 

Code Block
languagejava
public static final class CustomDynamicPartitioner implements DynamicPartitioner<byte[], Long> {
  @Override
  public PartitionKey getPartitionKey(byte[] key, Long value, long logicalStartTime) {
    return PartitionKey.builder().addLongField("time", logicalStartTime).addLongField("othervalue", value).build();
  }
}


 In the beforeSubmit() method of the user's MapReduce job, set the PartitionedFileSet dataset with the user's Dynamic Partitioner as output

...