User Stories 

• As a data pipeline user, I would like to be able to track data changes in Salesforce and load it into a sink.

...

When something is changed in entity we will receive notification event for it. 

Design

Properties:

• clientId: Client ID from the connected app

• clientSecret: Client Secret from the connected app

• username: Username

• password: Password

• loginUrl:(default consumerKey: Application Consumer Key. This is also known as the OAuth client id. A Salesforce connected application must be created in order to get a consumer key.

• consumerSecret: Application Consumer Secret. This is also known as the OAuth client secret. A Salesforce connected application must be created in order to get a client secret.

• username: Username to use when connecting to Salesforce.

• password: Password to use when connecting to Salesforce.

• loginUrl: Salesforce login URL to authenticate against. The default value ishttps://login.salesforce.com/services/oauth2/token) For Salesforce sandbox runs login url is different. That's why user needs this option.objects - list of object's API names . This should be changed when running against the Salesforce sandbox.

• objects - Objects to read change events from (For example: Task for base object and Employee__c for custom) separated by ",". If list is empty then subscription for all events will be used.

Handle errors - Possible values are: "Skip on error" or "Fail on error". These are strategies on handling records which cannot be transformed

• .

"Skip on error" - just skip, "Fail on error" - fails the pipeline if at least one erroneous record is found.

Example JSON

{
        "name": "CDCSalesforce",
        "plugin": {
        "name": "CDCSalesforce",
        "type": "realtimesource",
        "label": "CDCSalesforce",
        "properties": {
           "clientIdconsumerKey": "XXXXXXX",
           "clientSecretconsumerSecret": "XXXXXXXXX",
           "username": "XXXXX",
           "password": "XXXXX",               
           "loginUrl": "https://login.salesforce.com/services/oauth2/token",
		   "objects": "XXXXX,XXXXX",
		   "handleErrors": "Skip on error"
         }
       }
 }

DML

We subscribe for appropriate topics to get events for entities when something changes in database.

...

Limitations

• Offset for event's topic is not supported
• Transactional operations are not supported yet

Implementation

Salesforce CDC API will be used for realtime plugin.

Cometd BayeuxClient is used to subscribe to events.