...
...
...
...
...
Introduction
The SendGrid is a communication platform for transactional and marketing emails.
User
...
- User will retrieve API Keys (Restricted Access) from their SendGrid account for Event Notification, Marketing Campaigns, Suppressions and Tracking.
- User will enable Active Tracking in their account - Click, Open and Subscription tracking is enabled.
- Uses SendGrid REST APIs and Webhooks to sync your data.
- Retrieves
- Campaigns,
- Senders,
- Lists,
- Recipients,
- Unsubscribe groups,
- Advanced Stats,
- Bounces,
- Category Stats,
- Global Stats,
- Lists,
- Marketing Campaigns,
- Schedule,
- Segments,
User Configurations
Storie(s)
- As a pipeline developer, I would like to query analytical data for marketing emails sent through Sendgrid, so that I can analyze and combine that data with other data sources.
- As a pipeline developer, I would like to bulk send marketing emails to target customers through Sendgrid, so that I can effectively target them in marketing campaigns.
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Batch source
Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget | |
|---|---|---|---|---|---|---|---|
| Basic | Authentication type | The way, how user would like to be authenticated to the SendGrid acoount | Basic Api Key | Basic | auth-type | radio-group | |
| API Key | The SendGrid API Key taken from the SendGrid account of choice | Shows if "Authentication type" is set to "api" | sendgrid-api-key | Text Boxsecurekey-text | |||
| Username | Login name for | SendgridSendGrid | Shows if "Authentication type" is set to "basic" | username | Text Boxtextbox | ||
| Password | Login password for the username specified abovePassword for the SendGrid account | Shows if "Authentication type" is set to "basic" | password | Text Boxpassword | |||
| Data Source Types | List of data source groups |
| dataSourceTypes | multi-select | |||
| Marketing Campaign Objects | One of the above sources picked from list | data-source | Text Box Marketing Campaigns Fields
| dataSourceMarketing | multi-select | ||
| Statistic Objects | One of the above sources picked from list | Stats
| dataSourceStats | multi-select | |||
| Suppression Objects | One of the above sources picked from list | Suppressions
| dataSourceSuppressions | multi-select | |||
| Data Source Fields | List of fields to be retrieved | see "Data sources description" section below | dataSourceFields | multi-select | |||
| Options | Start Date | (required by some SG API Endpoints) The date in format YYYY-MM-DD, starting from which the data | -source-fieldsText Boxis requested. | Displayed depends on list of the selected data sources | not set | start-date | textbox |
| End Date | (required by some SG API Endpoints) The date in format YYYY-MM-DD, the end date for the requested data | Displayed depends on list of the selected data sources | not set | end-date | textbox | ||
| Error Handling | On Record Error | How to handle error in record processing |
| Skip error | on-error | Radio Button (layout: block) |
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Batch sink
Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|
| Basic | Authentication type | The way, how user would like to be authenticated to the SendGrid account | Possible options:
Required field | Basic | auth-type | radio-group |
| API Key | The SendGrid API Key taken from the SendGrid account | Shows if "Authentication type" is set to "api" | sendgrid-api-key | securekey-text | ||
| Username | Login name for the SendGrid account | Shows if "Authentication type" is set to "basic" | username | textbox | ||
| Password | Password for the SendGrid account | Shows if "Authentication type" is set to "basic" | password | password | ||
| From | The author of the message | Required field | from | textbox | ||
| Recipient address source | Recipients addresses source selection | The recipient address possible sources:
Would be displayed depends on selection:
Required field | recipientAddressSource | radio-group | ||
Column name for recipients addresses | Name of the column with coma-separated list of recipients | recipientColumnName | textbox | |||
| Recipient address list | List of mail recipients | recipientConfigAddressList | csv | |||
| Email subject | Email Subject | Required field | mailSubject | textbox | ||
| Email body column name | Name of the column for the mail content | Required field | bodyColumnName | textbox | ||
| Reply To | Email address to which the author of the message suggests that replies be sent | replyTo | textbox | |||
| Options | Custom Mail Footer | Footer feature setting switcher | Possible states:
| Off | footerEnabled | toggle |
| Footer HTML | The default footer which would be included to every email | footerHtml | textarea | |||
| Sandbox Mode | Allows to send a test email to ensure that your request body is valid and formatted correctly | Possible states:
| Off | sandboxMode | toggle | |
| Click Tracking | Allows to track whether a recipient clicked a link in a email | Possible states:
| Off | clickTracking | toggle | |
| Open Tracking | Allows to track whether the email was opened or not, by including a single pixel image in the body of the content. When the pixel is loaded, SendGrid can log that the email was opened | Possible states:
| Off | openTracking | toggle | |
| Subscription Tracking | Allows to insert a subscription management link at the bottom of the text and html bodies of an email | Possible states:
| Off | subscriptionTracking | toggle |
Design Tips and references
SendGrid v3 API:
- https://sendgrid.api-docs.io/v3.0/
- https://sendgrid.com/docs/API_Reference/Webapi_API_v3/index.html
SendGrid supports API v2 and v3 with Legacy Marketing Campaigns (LMC) API and New Marketing Campaigns(NMC) legacy and and SendGrid v3 API . As v2 and LMC API v3 legacy are outdated, plugin we going to support and use the latest SendGrid v3 API.
User Expectations
- User will retrieve API Keys (Restricted Access) from their SendGrid account for Event Notification, Marketing Campaigns, Suppressions and Tracking.
- User will enable Active Tracking in their account - Click, Open and Subscription tracking is enabled.
- Uses SendGrid v3
...
- REST API and Webhooks to sync your data.
- Retrieves
- Marketing Campaigns
- Automation
- Single Sends
- Senders
- Contacts
- Segments
- Schedule *
- Stats
- Global Stats
- Category Stats
- Advanced Stats
- Suppressions
- Bounces
- Global Unsubscribes
- Group Unsubscribes
- Spam Reports **
- Blocks **
* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks - Marketing Campaigns
Data source reference
| Object | API Reference | API Premissions | Notes | CampaignsNotes | ||
|---|---|---|---|---|---|---|
| Automations | - | No documentation available, but the usage could be revers engineered through debugging of official UI | ||||
| Single Sends | https://sendgrid.api-docs.io/v3.0/listssingle-sends/get-all-lists | * | Senders-single-sends | |||
| Senders | - | No documentation available, but the usage could be revers engineered through debugging of official UI | ||||
| Contacts | https://sendgrid.api-docs.io/v3.0/senderscontacts/createget-a-sender-identity-1 | v3 API allow to create sender identities, still looking for retrieving existing ones | Lists | all-contacts | ||
| Segments | https://sendgrid.api-docs.io/v3.0/listssegmenting-contacts/get-list-allof-listssegments | * | ||||
| RecipientsGlobalStats | https://sendgrid.api-docs.io/v3.0/contactsstats/getretrieve-global-allemail-contactsstatistics | |||||
| Unsubscribe groupCategoryStats | https://sendgrid.api-docs.io/v3.0/suppressions-unsubscribe-groupscategories/retrieve-allemail-suppressionstatistics-groups-associated-with-the-user | Advanced Stats | for-categories | |||
| AdvancedStats | https://sendgrid.com/docs/API_Reference/Web_API_v3/Stats/advanced.html | |||||
| Bounces | https://sendgrid.api-docs.io/v3.0/bounces-api/retrieve-all-bouncesCategory Stats | |||||
| Global Unsubscribes | - | No documentation available, but the usage could be revers engineered through debugging of official UI | ||||
| Group Unsubscribes | https://sendgrid.api-docs.io/v3.0/categoriessuppressions-unsubscribe-groups/retrieve-email-statistics-for-categoriesGlobal Statsall-suppression-groups-associated-with-the-user | |||||
| Spam Reports | https://sendgrid.api-docs.io/v3.0/statsspam-reports-api/retrieve-globalall-emailspam-statisticsreports | |||||
| Marketing CampaignsBlocks | https://sendgrid.api-docs.io/v3.0/singleblocks-sendsapi/get-all-single-sends | Schedule | TODO | Segments | https://sendgrid.api-docs.io/v3.0/segmenting-contacts/get-list-of-segmentsretrieve-all-blocks |
Data sources description
| Field name | Type |
|---|---|
Automation | |
| id | string |
| name | string |
| status | string |
| type | string |
| message_count | number |
| created_at | string |
| updated_at | string |
| live_at | string |
SingleSends | |
| id | string |
| name | string |
| status | string |
| created_at | string |
| updated_at | string |
| is_abtest | boolean |
Senders | |
| id | string |
| nickname | string |
| address | string |
| address_2 | string |
| city | string |
| country | string |
| state | string |
| zip | number |
| locked | boolean |
| created_at | string |
| updated_at | string |
| from_email | string |
| from_name | string |
| verified_status | string |
| verified_reason | string |
| reply_to_name | string |
| reply_to_email | string |
Contacts | |
| id | string |
| first_name | string |
| last_name | string |
| list_ids | list[string] |
| created_at | string |
| updated_at | string |
| string | |
Segments | |
| id | string |
| name | string |
| parent_list_id | string |
| created_at | datetime |
| updated_at | datetime |
| sample_updated_at | datetime |
| contacts_count | number |
GlobalStats | |
| date | string |
| metrics_blocks | number |
| metrics_bounce_drops | number |
| metrics_bounces | number |
| metrics_clicks | number |
| metrics_deferred | number |
| metrics_invalid_emails | number |
| metrics_opens | number |
| metrics_processed | number |
| metrics_requests | number |
| metrics_spam_report_drops | number |
| metrics_spam_reports | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
| metrics_unsubscribe_drops | number |
| metrics_unsubscribes | number |
CategoryStats | |
| name | string |
| type | string |
| date | string |
| metrics_blocks | number |
| metrics_bounce_drops | number |
| metrics_bounces | number |
| metrics_clicks | number |
| metrics_deferred | number |
| metrics_delivered | number |
| metrics_invalid_emails | number |
| metrics_opens | number |
| metrics_processed | number |
| metrics_requests | number |
| metrics_spam_report_drops | number |
| metrics_spam_reports | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
| metrics_unsubscribe_drops | number |
| metrics_unsubscribes | number |
AdvancedStats | |
| name | string |
| type | string |
| date | string |
| metrics_clicks | number |
| metrics_opens | number |
| metrics_unique_clicks | number |
| metrics_unique_opens | number |
Bounces | |
| created | string |
| string | |
| reason | string |
| status | string |
GlobalUnsubscribes | |
| created | string |
| string | |
GroupUnsubscribes | |
| id | string |
| name | string |
| description | string |
| is_default | boolean |
| last_email_send_at | string |
| unsubscribes | number |
Authorization
Plugin supporting Basic and API Key authorization, with using API Key by default when provided.
Rate limiting
SendGrid v3 API have an request rate limiting. The API requester have integrated way to retry over time to fetch wait for penalty time.
Approach
Create a module sendgrid plugin in sendgrid project with use of existing SendGrid library. Add SendGrid objects with their deserialization from API and transformation to structured records.
UI Impact or Changes
SendGrid source and sink are separate entries with SendGrid logo in source and sink lists.
Test Case(s)
TODO
Sample Pipeline
TODO