...
An aggregator plugin will operate over windows in the stream, and can be implemented by a call to flatMapToPair(), then reduceByKey().
Windowing and Reduce can be implemented through a general StreamingTransform API:
...
Code Block |
---|
public class Window implements StreamTransform<StructuredRecord, StructuredRecord> { public JavaDStream<StructuredRecord> transform(JavaDStream<StructuredRecord> stream) { return stream.window(Durations.millis(conf.durationMS)); } } |
A plugin that computes the sum of a field would be implemented like:
Code Block |
---|
public class Sum implements StreamTransform<StructuredRecord, StructuredRecord> {
public JavaDStream<StructuredRecord> transform(JavaDStream<StructuredRecord> stream) {
stream.reduce(new Function2<StructuredRecord, StructuredRecord, StructuredRecord>() {
@Override
public StructuredRecord call(StructuredRecord v1, StructuredRecord v2) throws Exception {
return StructuredRecord.builder(conf.getSchema())
.set(conf.sumField, v1.get(conf.sumField) + v2.get(conf.sumField))
.build();
}
});
}
} |
...