Page Comparison

Introduction 

Salesforce Sync plugins can help users automatically sync their Salesforce objects to specified destinations with only a few clicks.

User Stories

• As a data pipeline user, I would like to be able to create a pipeline to sync a single Salesforce object to a specified destination, so that I can use the Salesforce object for analytics in my downstream processes.
• As a data pipeline user, I would like to be able to create a pipeline to sync multiple Salesforce objects to their corresponding destinations, so that I can use the Salesforce objects for analytics in my downstream processes.

Requirements

• Support BigQuery and Cloud SQL as destinations
• Select source object(s) and a destination. For each object, create a corresponding object in the destination (if it doesn't exist already), and push data to it. 
• A user should be able to easily identify the destination object based on its name. The name should be unique, and must contain the source object name. For uniqueness purposes, more information may be encoded in the destination object name.
• Support full (get all data since the beginning of time) and incremental (get data since the specified timestamp) modes. 
• Automatically handle schema changes between source and destination. If a destination exists, but with a different schema from the source because the source's schema has been updated, the destination's schema should be updated.
  

User flow

1. User specifies Salesforce credentials (username, password, clientId and clientSecret)
   1. The source plugin determines the Salesforce instance to use based on these credentials
2. User selects the Salesforce objects he wants to replicate
   1. There may be a limitation here that CDAP currently does not have the capability to list all objects for the user to select from. If that's the case, we may require the user to manually enter the objects to begin with, and then enhance this experience later (perhaps when connections and plugins are unified)
3. User specifies whether he wants to run a "full refresh" (pull all data since the beginning of time) or incremental (pull data since the specified timestamp).
4. User specifies the destination. 
   
   1. This is perhaps a different set of sink plugins
   2. There is one such plugin for GCS, Cloud SQL, BigQuery and Spanner
   3. For each such plugin, the user only specifies the top level container. e.g. For GCS, user specifies a bucket. For BigQuery, a dataset, for CloudSQL, a database, and so on.
5. For all Salesforce objects selected
• The sink plugins create a corresponding object in the destination- e.g. a directory in the GCS bucket, a table in the BigQuery dataset, a table in the CloudSQL database.
• The sink plugins ensure that the schema of the destination objects is consistent with the schema of the source, at the time of the run

Type of plugin

1. Batch Source
2. Batch Sink (s)

Technical Design

Source Properties

• clientId: Client ID from the connected app 
• cllientSecret: Client Secret from the connected app
• username: Username
• password: Password
• loginUrl (default is https://login.salesforce.com/services/oauth2/token) For Salesforce sandbox runs login url is different. That's why user needs this option.
• objects: list of Salesforce objects to replicate

Sink Properties

BigQuery

• (Optional) Project ID
• (Optional) Service Account File Path
• (Required) Dataset