...
- Pluggable stream record formats (the format in which data is read from a stream, which is different from the format in which files are written to a stream)
- Expose cdap-spi module that contains StreamEventRecordFormat abstract class
- Each StreamEventRecordFormat will be associated with a simple name (e.g. grok, clf, avro)
- "system" record formats will come from within the CDAP codebase (grok, clf, avro)
- "user" record formats will be loaded from jars in a certain directory containing SPI jars
- In a later revision, this will may be namespaced and/or managed via an HTTP API
- Stream views
- A stream view is an explorable view (Hive table) of a stream, with a particular record format
- A stream may have multiple views
- Upon creating a stream, the stream will have a default view
Stream View HTTP API
Note: Moved to Views
Changes to existing APIs
| Path | Request | Response | Notes |
|---|---|---|---|
| PUT /v3/namespaces/<namespace>/streams/<stream> | Instead of creating a Hive table with a default record format, this will create a "default" view with a default record format. | ||
| DELETE /v3/namespaces/<namespace>/streams/<stream> | This will delete all associated views for the stream. | ||
| POST /v3/namespaces/<namespace>/streams/properties | "format" field will be considered "deprecated" -> if format is given, this modifies the default view for backwards compat | Notify user that "format" field is deprecated somehow? |
...