The salesforce batch sink is responsible for using Salesforce API to upsert salesforce objects. The sink should handle large batches of data (~10 GB) and should handle all object types - contacts, campaigns, oppurtunities, leads, custom objects. Users should be able to upload subset of fields
Use Cases
As a ETL developer, I would like to consolidate contacts data from multiple source systems and update in Salesforce
- As a ETL developer, I would like to aggregate results of a campaign from data warehouse into Salesforce
Assumptions
- Compound fields such as address and geo location will be represented as nested records in the source system
User Expectations
- Users should be able configure the login URL
- Hints should be provide in the default value
- Users should be able configure the sensitive information such as client id, secret in secure store
- Any error in the upload fails the pipeline
- Field level metadata should be captured by the sink
User Configurations
| Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|
| Standard | Login REST URL | A REST API for Salesforce cloud | salesforce-rest-url | Text Box | ||
| Object Type | Type of Salesforce object - ex: contacts, campaigns, oppurtunities | record-type | Drop down | |||
| Username | Salesforce username | username | Text Box | |||
| Shared Secret | Salesforce password | password | Text Box | |||
| Advanced | Batch Size | Batch size of the upload | batch-size | Text Box |