User Stories 

• As a data pipeline user, I would like to be able to track data changes in Salesforce and load it into a sink.

...

When something is changed in entity we will receive notification event for it. 

Design

Properties:

• clientId: Client ID from the connected app

• cllientSecret: Client Secret from the connected app

• username: Username

• password: Password

• loginUrl (default consumerKey: Application Consumer Key. This is also known as the OAuth client id. A Salesforce connected application must be created in order to get a consumer key.

• consumerSecret: Application Consumer Secret. This is also known as the OAuth client secret. A Salesforce connected application must be created in order to get a client secret.

• username: Username to use when connecting to Salesforce.

• password: Password to use when connecting to Salesforce.

• loginUrl: Salesforce login URL to authenticate against. The default value ishttps://login.salesforce.com/services/oauth2/token) For Salesforce sandbox runs login url is different. That's why user needs this option.objects - list of object's API names . This should be changed when running against the Salesforce sandbox.

• objects - Objects to read change events from (For example: Task for base object and Employee__c for custom) separated by ",". If list is empty then subscription for all events will be used.

• Event offset - Possible values are: "Earliest" or "Latest" (default). "Earliest" allows to get event for last 3 days (Salesforce stores events only for last 3 days), "Latest" subscribes for new events.
• Handle errors - Possible values are: "Skip on error" or "Fail on error". These are strategies on handling records which cannot be transformed. "Skip on error" - just skip, "Fail on error" - fails the pipeline if at least one erroneous record is found.

Example JSON

{
        "name": "CDCSalesforce",
        "plugin": {
        "name": "CDCSalesforce",
        "type": "realtimesource",
        "label": "CDCSalesforce",
        "properties": {
           "clientIdconsumerKey": "XXXXXXX",
           "clientSecretconsumerSecret": "XXXXXXXXX",
           "username": "XXXXX",
           "password": "XXXXX",               
           "loginUrl": "https://login.salesforce.com/services/oauth2/token",
		   "objects": "XXXXX,XXXXX",
           "eventOffset": "Latest",
		 
 "handleErrors": "Skip on error"          }
       }
 }

DML

We subscribe for appropriate topics to get events for entities when something changes in database.

...

Then we will compare response with previous info about entity, if it exists, and send change event if necessary. If there is no previous info, than create event will be sent.

Limitations

• Offset for event's topic is not supported
• Transactional operations are not supported yet

Implementation

Salesforce CDC API will be used for realtime plugin.

Cometd BayeuxClient is used to subscribe to events.