...
A Join stage with n inputs will implemented as n mapToPair() calls on all inputs into the stage, then x - 1 joins (assuming x inputs).
Windowing and Reduce can be implemented through a general StreamingTransform API:
| Code Block |
|---|
public interface StreamTransform<IN, OUT> {
public JavaDStream<OUT> transform(JavaDStream<IN> stream) throws Exception;
} |
For example, a window plugin would be implemented like:
| Code Block |
|---|
public class Window implements StreamTransform<StructuredRecord, StructuredRecord> {
public JavaDStream<StructuredRecord> transform(JavaDStream<StructuredRecord> stream) {
return stream.window(Durations.millis(conf.durationMS));
}
} |
A plugin that computes the sum of a field would be implemented like:
| Code Block |
|---|
public class Sum implements StreamTransform<StructuredRecord, StructuredRecord> {
public JavaDStream<StructuredRecord> transform(JavaDStream<StructuredRecord> stream) {
stream.reduce(new Function2<StructuredRecord, StructuredRecord, StructuredRecord>() {
@Override
public StructuredRecord call(StructuredRecord v1, StructuredRecord v2) throws Exception {
return StructuredRecord.builder(conf.getSchema())
.set(conf.sumField, v1.get(conf.sumField) + v2.get(conf.sumField))
.build();
}
});
}
} |