...

Introduction

The SendGrid is a communication platform for transactional and marketing emails.

User

...

Automation
Single Sends
Senders
Contacts
Segments
Schedule *

...

Global Stats
Category Stats
Advanced Stats

...

Bounces
Global Unsubscribes
Group Unsubscribes
Spam Reports **
Blocks **

* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks

User Configurations

...

Section

...

User Configuration Label

...

Label Description

...

Options

...

Default

...

Variable

...

User Widget

...

Storie(s)

As a pipeline developer, I would like to query analytical data for marketing emails sent through Sendgrid, so that I can analyze and combine that data with other data sources.
As a pipeline developer, I would like to bulk send marketing emails to target customers through Sendgrid, so that I can effectively target them in marketing campaigns.

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

Configurables

Batch source

SendgridLogin password for the username specified above

Section	User Configuration Label	Label Description	Options	Default	Variable	User Widget
Basic	Authentication type	The way, how user would like to be authenticated to the SendGrid acoount	Basic Api Key	Basic	auth-type	radio-group
	API Key	The SendGrid API Key taken from the SendGrid account of choice	Shows if "Authentication type" is set to "api"		sendgrid-api-key	Text Boxsecurekey-text
	Username	Login name for	SendGrid	Shows if "Authentication type" is set to "basic"		username	Text Boxtextbox
	Password	Password for the SendGrid account	Shows if "Authentication type" is set to "basic"		password	Text Boxpassword
	Data Source Types	List of data source groups	Marketing Campaigns Fields Stats Fields Suppressions Fields		dataSourceTypes	multi-select
	Marketing Campaign Objects	One of the above sources picked from list	data-source	Text Box Marketing Campaigns Fields Automation Single Sends Senders Contacts Segments		dataSourceMarketing	multi-select
	Statistic Objects	One of the above sources picked from list	Stats Global Stats Category Stats Advanced Stats		dataSourceStats	multi-select
	Suppression Objects	One of the above sources picked from list	Suppressions Bounces Global Unsubscribes Group Unsubscribes		dataSourceSuppressions	multi-select
	Data Source Fields	List of fields to be retrieved	data-source-fields	Text Boxsee "Data sources description" section below		dataSourceFields	multi-select

Options	Start Date	(required by some SG API Endpoints) The date in format YYYY-MM-DD, starting from which the data is requested.	Displayed depends on list of the selected data sources	not set	start-date	TODOtextbox
	End Date	(required by some SG API Endpoints) The date in format YYYY-MM-DD, the end date for the requested data	Displayed depends on list of the selected data sources	not set	end-date	TODOtextbox
Error Handling	On Record Error	How to handle error in record processing	Skip error Send to error port Fail pipeline	Skip error	on-error	Radio Button (layout: block)

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

Batch sink

Section	User Configuration Label	Label Description	Options	Default	Variable	User Widget
Basic	Authentication type	The way, how user would like to be authenticated to the SendGrid account	Possible options: Basic Api Key Required field	Basic	auth-type	radio-group
	API Key	The SendGrid API Key taken from the SendGrid account	Shows if "Authentication type" is set to "api"		sendgrid-api-key	securekey-text
	Username	Login name for the SendGrid account	Shows if "Authentication type" is set to "basic"		username	textbox
	Password	Password for the SendGrid account	Shows if "Authentication type" is set to "basic"		password	password
	From	The author of the message	Required field		from	textbox
	Recipient address source	Recipients addresses source selection	The recipient address possible sources: Input Record (input) Configuration (config) Would be displayed depends on selection: recipientColumnName recipientConfigAddressList Required field		recipientAddressSource	radio-group
	Column name for recipients addresses	Name of the column with coma-separated list of recipients			recipientColumnName	textbox
	Recipient address list	List of mail recipients			recipientConfigAddressList	csv
	Email subject	Email Subject	Required field		mailSubject	textbox
	Email body column name	Name of the column for the mail content	Required field		bodyColumnName	textbox
	Reply To	Email address to which the author of the message suggests that replies be sent			replyTo	textbox
Options	Custom Mail Footer	Footer feature setting switcher	Possible states: On (true) Off (false)	Off	footerEnabled	toggle
	Footer HTML	The default footer which would be included to every email			footerHtml	textarea
	Sandbox Mode	Allows to send a test email to ensure that your request body is valid and formatted correctly	Possible states: On (true) Off (false)	Off	sandboxMode	toggle
	Click Tracking	Allows to track whether a recipient clicked a link in a email	Possible states: On (true) Off (false)	Off	clickTracking	toggle
	Open Tracking	Allows to track whether the email was opened or not, by including a single pixel image in the body of the content. When the pixel is loaded, SendGrid can log that the email was opened	Possible states: On (true) Off (false)	Off	openTracking	toggle
	Subscription Tracking	Allows to insert a subscription management link at the bottom of the text and html bodies of an email	Possible states: On (true) Off (false)	Off	subscriptionTracking	toggle

Design Tips and references

...

SendGrid supports API v2 and v3 legacy and and SendGrid v3 API . As v2 and v3 legacy are outdated, we going to support and use the latest SendGrid v3 API.

User Expectations

User will retrieve API Keys (Restricted Access) from their SendGrid account for Event Notification, Marketing Campaigns, Suppressions and Tracking.
User will enable Active Tracking in their account - Click, Open and Subscription tracking is enabled.
Uses SendGrid v3 REST API and Webhooks to sync your data.
Retrieves
- Marketing Campaigns
  - Automation
  - Single Sends
  - Senders
  - Contacts
  - Segments
  - Schedule *
- Stats
  - Global Stats
  - Category Stats
  - Advanced Stats
- Suppressions
  - Bounces
  - Global Unsubscribes
  - Group Unsubscribes
  - Spam Reports **
  - Blocks **
* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks

Data source reference

Object	API Reference	API Premissions	Notes
Automations	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Single Sends	https://sendgrid.api-docs.io/v3.0/single-sends/get-all-single-sends
Senders	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Contacts	https://sendgrid.api-docs.io/v3.0/contacts/get-all-contacts
Segments	https://sendgrid.api-docs.io/v3.0/segmenting-contacts/get-list-of-segments
GlobalStats	https://sendgrid.api-docs.io/v3.0/stats/retrieve-global-email-statistics
CategoryStats	https://sendgrid.api-docs.io/v3.0/categories/retrieve-email-statistics-for-categories
AdvancedStats	https://sendgrid.com/docs/API_Reference/Web_API_v3/Stats/advanced.html
Bounces	https://sendgrid.api-docs.io/v3.0/bounces-api/retrieve-all-bounces
Global Unsubscribes	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Group Unsubscribes	https://sendgrid.api-docs.io/v3.0/suppressions-unsubscribe-groups/retrieve-all-suppression-groups-associated-with-the-user
Spam Reports	https://sendgrid.api-docs.io/v3.0/spam-reports-api/retrieve-all-spam-reports
Blocks	https://sendgrid.api-docs.io/v3.0/blocks-api/retrieve-all-blocks	Schedule	-

Data sources

...

description

Schedule (TODO)

Field name	Type	Description
Automation
Single Sends
Senders
Contacts
Segments
Global Stats
Category Stats
Advanced Stats
Bounces
Global Unsubscribes
Group Unsubscribes
Spam Reports (TODO)
Blocks (TODO)

Automation
id	string
name	string
status	string
type	string
message_count	number
created_at	string
updated_at	string
live_at	string
SingleSends
id	string
name	string
status	string
created_at	string
updated_at	string
is_abtest	boolean
Senders
id	string
nickname	string
address	string
address_2	string
city	string
country	string
state	string
zip	number
locked	boolean
created_at	string
updated_at	string
from_email	string
from_name	string
verified_status	string
verified_reason	string
reply_to_name	string
reply_to_email	string
Contacts
id	string
first_name	string
last_name	string
list_ids	list[string]
created_at	string
updated_at	string
email	string
Segments
id	string
name	string
parent_list_id	string
created_at	datetime
updated_at	datetime
sample_updated_at	datetime
contacts_count	number
GlobalStats
date	string
metrics_blocks	number
metrics_bounce_drops	number
metrics_bounces	number
metrics_clicks	number
metrics_deferred	number
metrics_invalid_emails	number
metrics_opens	number
metrics_processed	number
metrics_requests	number
metrics_spam_report_drops	number
metrics_spam_reports	number
metrics_unique_clicks	number
metrics_unique_opens	number
metrics_unsubscribe_drops	number
metrics_unsubscribes	number
CategoryStats
name	string
type	string
date	string
metrics_blocks	number
metrics_bounce_drops	number
metrics_bounces	number
metrics_clicks	number
metrics_deferred	number
metrics_delivered	number
metrics_invalid_emails	number
metrics_opens	number
metrics_processed	number
metrics_requests	number
metrics_spam_report_drops	number
metrics_spam_reports	number
metrics_unique_clicks	number
metrics_unique_opens	number
metrics_unsubscribe_drops	number
metrics_unsubscribes	number
AdvancedStats
name	string
type	string
date	string
metrics_clicks	number
metrics_opens	number
metrics_unique_clicks	number
metrics_unique_opens	number
Bounces
created	string
email	string
reason	string
status	string
GlobalUnsubscribes
created	string
email	string
GroupUnsubscribes
id	string
name	string
description	string
is_default	boolean
last_email_send_at	string
unsubscribes	number

Authorization

Plugin supporting Basic and API Key authorization, with using API Key by default when provided.

Rate limiting

SendGrid v3 API have an request rate limiting. The API requester have integrated way to retry over time to fetch wait for penalty time.

Approach

Create a module sendgrid plugin in sendgrid project with use of existing SendGrid library. Add SendGrid objects with their deserialization from API and transformation to structured records.

UI Impact or Changes

SendGrid source and sink are separate entries with SendGrid logo in source and sink lists.

Test Case(s)

TODO

Sample Pipeline

TODO

Page Comparison

Versions Compared

Old Version 11

New Version Current

Key

<span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-0">[data-colorid=</span>

User

User Configurations

Storie(s)

Plugin Type

Configurables

Batch source

Plugin Type

Batch sink

Design Tips and references

User Expectations

Data source reference

Data sources

description

Automation

Single Sends

Senders

Contacts

Segments

Global Stats

Category Stats

Advanced Stats

Bounces

Global Unsubscribes

Group Unsubscribes

Spam Reports (TODO)

Blocks (TODO)

Automation

SingleSends

Senders

Contacts

Segments

GlobalStats

CategoryStats

AdvancedStats

Bounces

GlobalUnsubscribes

GroupUnsubscribes

Authorization

Rate limiting

Approach

UI Impact or Changes

Test Case(s)

Sample Pipeline