SendGrid
- Nitin Motgi
- Dmytro Grinenko
- Bhooshan Mogal
Introduction
The SendGrid is a communication platform for transactional and marketing emails.
User Storie(s)
- As a pipeline developer, I would like to query analytical data for marketing emails sent through Sendgrid, so that I can analyze and combine that data with other data sources.
- As a pipeline developer, I would like to bulk send marketing emails to target customers through Sendgrid, so that I can effectively target them in marketing campaigns.
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Batch source
Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|
| Basic | Authentication type | The way, how user would like to be authenticated to the SendGrid acoount | Basic Api Key | Basic | auth-type | radio-group |
| API Key | The SendGrid API Key taken from the SendGrid account of choice | Shows if "Authentication type" is set to "api" | sendgrid-api-key | securekey-text | ||
| Username | Login name for SendGrid | Shows if "Authentication type" is set to "basic" | username | textbox | ||
| Password | Password for the SendGrid account | Shows if "Authentication type" is set to "basic" | password | password | ||
| Data Source Types | List of data source groups |
| dataSourceTypes | multi-select | ||
| Marketing Campaign Objects | One of the above sources picked from list | Marketing Campaigns Fields
| dataSourceMarketing | multi-select | ||
| Statistic Objects | One of the above sources picked from list | Stats
| dataSourceStats | multi-select | ||
| Suppression Objects | One of the above sources picked from list | Suppressions
| dataSourceSuppressions | multi-select | ||
| Data Source Fields | List of fields to be retrieved | see "Data sources description" section below | dataSourceFields | multi-select | ||
| Options | Start Date | (required by some SG API Endpoints) The date in format YYYY-MM-DD, starting from which the data is requested. | Displayed depends on list of the selected data sources | not set | start-date | textbox |
| End Date | (required by some SG API Endpoints) The date in format YYYY-MM-DD, the end date for the requested data | Displayed depends on list of the selected data sources | not set | end-date | textbox | |
| Error Handling | On Record Error | How to handle error in record processing |
| Skip error | on-error | Radio Button (layout: block) |
Batch sink
Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|
| Basic | Authentication type | The way, how user would like to be authenticated to the SendGrid account | Possible options:
Required field | Basic | auth-type | radio-group |
| API Key | The SendGrid API Key taken from the SendGrid account | Shows if "Authentication type" is set to "api" | sendgrid-api-key | securekey-text | ||
| Username | Login name for the SendGrid account | Shows if "Authentication type" is set to "basic" | username | textbox | ||
| Password | Password for the SendGrid account | Shows if "Authentication type" is set to "basic" | password | password | ||
| From | The author of the message | Required field | from | textbox | ||
| Recipient address source | Recipients addresses source selection | The recipient address possible sources:
Would be displayed depends on selection:
Required field | recipientAddressSource | radio-group | ||
Column name for recipients addresses | Name of the column with coma-separated list of recipients | recipientColumnName | textbox | |||
| Recipient address list | List of mail recipients | recipientConfigAddressList | csv | |||
| Email subject | Email Subject | Required field | mailSubject | textbox | ||
| Email body column name | Name of the column for the mail content | Required field | bodyColumnName | textbox | ||
| Reply To | Email address to which the author of the message suggests that replies be sent | replyTo | textbox | |||
| Options | Custom Mail Footer | Footer feature setting switcher | Possible states:
| Off | footerEnabled | toggle |
| Footer HTML | The default footer which would be included to every email | footerHtml | textarea | |||
| Sandbox Mode | Allows to send a test email to ensure that your request body is valid and formatted correctly | Possible states:
| Off | sandboxMode | toggle | |
| Click Tracking | Allows to track whether a recipient clicked a link in a email | Possible states:
| Off | clickTracking | toggle | |
| Open Tracking | Allows to track whether the email was opened or not, by including a single pixel image in the body of the content. When the pixel is loaded, SendGrid can log that the email was opened | Possible states:
| Off | openTracking | toggle | |
| Subscription Tracking | Allows to insert a subscription management link at the bottom of the text and html bodies of an email | Possible states:
| Off | subscriptionTracking | toggle |
Design Tips and references
SendGrid v3 API:
- https://sendgrid.api-docs.io/v3.0/
- https://sendgrid.com/docs/API_Reference/api_v3.html
SendGrid supports API v2 and v3 legacy and and SendGrid v3 API . As v2 and v3 legacy are outdated, we going to support and use the latest SendGrid v3 API.
User Expectations
- User will retrieve API Keys (Restricted Access) from their SendGrid account for Event Notification, Marketing Campaigns, Suppressions and Tracking.
- User will enable Active Tracking in their account - Click, Open and Subscription tracking is enabled.
- Uses SendGrid v3 REST API and Webhooks to sync your data.
- Retrieves
- Marketing Campaigns
- Automation
- Single Sends
- Senders
- Contacts
- Segments
- Schedule *
- Stats
- Global Stats
- Category Stats
- Advanced Stats
- Suppressions
- Bounces
- Global Unsubscribes
- Group Unsubscribes
- Spam Reports **
- Blocks **
* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks - Marketing Campaigns
Data source reference
Data sources description
| Field name | Type |
|---|---|
Automation | |
| id | string |
| name | string |
| status | string |
| type | string |
| message_count | number |
| created_at | string |
| updated_at | string |
| live_at | string |
SingleSends | |
| id | string |
| name | string |
| status | string |
| created_at | string |
| updated_at | string |
| is_abtest | boolean |
Senders | |
| id | string |
| nickname | string |
| address | string |
| address_2 | string |
| city | string |
| country | string |
| state | string |
| zip | number |
| locked | boolean |
| created_at | string |
| updated_at | string |
| from_email | string |
| from_name | string |
| verified_status | string |
| verified_reason | string |
| reply_to_name | string |
| reply_to_email | string |
Contacts | |
| id | string |
| first_name | string |
| last_name | string |
| list_ids | list[string] |
| created_at | string |
| updated_at | string |
| string | |
Segments | |
| id | string |
| name | string |
| parent_list_id | string |
| created_at | datetime |
| updated_at | datetime |
| sample_updated_at | datetime |
| contacts_count | number |
GlobalStats | |
| date | string |
| metrics_blocks | number |
| metrics_bounce_drops | number |
| metrics_bounces | number |
| metrics_clicks | number |
| metrics_deferred | number |
| metrics_invalid_emails | number |
| metrics_opens | number |
| metrics_processed | number |
| metrics_requests | number |
| metrics_spam_report_drops | number |
| metrics_spam_reports | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
| metrics_unsubscribe_drops | number |
| metrics_unsubscribes | number |
CategoryStats | |
| name | string |
| type | string |
| date | string |
| metrics_blocks | number |
| metrics_bounce_drops | number |
| metrics_bounces | number |
| metrics_clicks | number |
| metrics_deferred | number |
| metrics_delivered | number |
| metrics_invalid_emails | number |
| metrics_opens | number |
| metrics_processed | number |
| metrics_requests | number |
| metrics_spam_report_drops | number |
| metrics_spam_reports | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
| metrics_unsubscribe_drops | number |
| metrics_unsubscribes | number |
AdvancedStats | |
| name | string |
| type | string |
| date | string |
| metrics_clicks | number |
| metrics_opens | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
Bounces | |
| created | string |
| string | |
| reason | string |
| status | string |
GlobalUnsubscribes | |
| created | string |
| string | |
GroupUnsubscribes | |
| id | string |
| name | string |
| description | string |
| is_default | boolean |
| last_email_send_at | string |
| unsubscribes | number |
Authorization
Plugin supporting Basic and API Key authorization, with using API Key by default when provided.
Rate limiting
SendGrid v3 API have an request rate limiting. The API requester have integrated way to retry over time to fetch wait for penalty time.
Approach
Create a module sendgrid plugin in sendgrid project with use of existing SendGrid library. Add SendGrid objects with their deserialization from API and transformation to structured records.
UI Impact or Changes
SendGrid source and sink are separate entries with SendGrid logo in source and sink lists.
Test Case(s)
TODO
Sample Pipeline
TODO