...
Goal : Introduce the ability in CDAP Programs to create streams, datasets or register the use of Plugins.
Use Case:
- Outside of Adapters - In regular CDAP applications:
Say we want to write to write a stream that is used only in a Worker. Instead of adding the Stream in an Application level, we can look at
Status Quo: Currently, we have the ability to create streams, datasets only at CDAP Application level. And Plugins can be registered only in Adapters (through AdapterConfigurer). So if we want to remove the concept of Application Templates and Adapters, we have couple of options:
i) Introduce the ability to register plugins in Application's configure method.
...
- Worker:
public class SimpleWorker extends AbstractWorker {
@Override
public void configure() {
createDataset("etlrealtimestate", KeyValueTable.class);
addStream(new Stream("hello"));
addDatasetModule("abcModule", ABCModule.class);
usePlugin("realtimesource", "kafka", "source", PluginProperties.EMPTY);
usePlugin("realtimesink", "stream", "sink", PluginProperties.EMPTY);
}
} - MapReduce:
Very similar to what Worker looks like. - Flows and Flowlets:
public static class SimpleFlow extends AbstractFlow {
public void configureFlow() {
addFlowlet("abc", new SimpleFlowlet());
addStream(new Stream("hello"));
connectStream("hello", "abc");
}
}
public static class SimpleFlowlet extends AbstractFlowlet {
public
}public void configure() {
createDataset("data", KeyValueTable.class);
setName("SimpleFlowlet");
}
}