Use Cases:
- Validator Filter: All records of a transform that are invalid go into one dataset; the remainder go into another.
API:
Existing APIs (in MapReduceContext); sets a single Dataset as the output for the MapReduce job:
context.setOutput(String datasetName);
context.setOutput(String datasetName, Dataset dataset);
Additional APIs (in MapReduceContext); adds a Dataset to the set of output Datasets for the MapReduce job:
context.addOutput(String datasetName);
context.addOutput(String datasetName, Dataset dataset);
Example Usage:
public void beforeSubmit(MapReduceContext context) throws Exception {
context.setOutput();}
Approach:
Take an approach similar to org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.