...

Note that the config is different than in 3.2.x.  Name no longer refers to the plugin name, but is a unique name for that stage.  There is also a object dedicated to the plugin.

Developing a new Plugin

Testing (unit and integration) a new plugin should be as easy as possible.  Pain points today are with integrations with external systems like cassandra or elastic search.  Ideally its not hard to add them.  Maybe use Docker?

...

Unit test pain points are it's hard to test a source or sink in isolation, you end up copying table source or something like it into your plugin project.  We could have a etl-test module that provides mock sources and sinks to help with this.

Building the sdk and packages

...

For steps 5 and 6, we can add a maven profile that takes a property that is a path for additional artifacts to include.  It can copy those to the right place, then do whatever it is doing today.

Upgrading Hydrator

We are making backwards incompatible changes to the etl config. This will require a Hydrator upgrade, which will have its own upgrade process that is not tied to the cdap upgrade process.

The Hydrator upgrade tool will have a command that converts an old config into a new config. It will also have a command that will take cdap host/port/credential information and convert all 3.2.x pipelines into 3.3.x pipelines..3.x pipelines. 

New ETL Application features

...

Note: Flow requires plugin integration with flows.

Controlling Realtime source poll frequency

...

{
  "config": {
    "source": {
      "pollFrequency": "1s",
       ...
    }
  }
}

Jira LegacyserverCask Community Issue TrackerserverId45b48dee-c8d6-34f0-9990-e6367dc2fe4bkeyCDAP-3856 

Plugin configure time schema validation

...