...
The streamingsource plugin type simply takes a JavaSparkContext and returns a JavaDStream:
| Code Block |
|---|
public interfaceabstract class StreamingSource<T> implements PipelineConfigurable, Serializable { public abstract JavaDStream<T> getStream(JavaStreamingContext jssc); } |
...
| Code Block |
|---|
public class TextFileWatcherSource extends implements StreamingSource<StructuredRecord> { JavaDStream<StructuredRecord> getStream(JavaStreamingContext jssc) { return jssc.fileStream(conf.path, LongWritable.class, Text.class, TextInputFormat.class) .map(new TextToStructuredRecordFunction()); } } |
...
The streamingsink plugin type simply takes a JavaDStream and a context and performs some operation on it:
| Code Block |
|---|
public interfaceabstract class StreamingSink<T> implements PipelineConfigurable, Serializable { public abstract void write(JavaDStream<T> stream, JavaSparkExecutionContext jsec); } |
...
A reduce function plugin will reduce all the records in a window into a single record.
| Code Block |
|---|
public interfaceabstract ReduceFunction<T> implements Serializable { public abstract T reduce(T record1, T record2); } |
...
Windowing can be implemented through a general StreamingTransform API:
| Code Block |
|---|
public abstract interfaceclass StreamTransform<IN, OUT> implements PipelineConfigurable, Serializable { public abstract JavaDStream<OUT> transform(JavaDStream<IN> stream) throws Exception; } |
...
| Code Block |
|---|
public class Window implementsextends StreamTransform<StructuredRecord, StructuredRecord> { public JavaDStream<StructuredRecord> transform(JavaDStream<StructuredRecord> stream) { return stream.window(Durations.millis(conf.durationMS)); } } |