...

Introduction

The SendGrid is a communication platform for transactional and marketing emails.

User

...

Storie(s)

As a pipeline developer, I would like to query analytical data for marketing emails sent through Sendgrid, so that I can analyze and combine that data with other data sources.
As a pipeline developer, I would like to bulk send marketing emails to target customers through Sendgrid, so that I can effectively target them in marketing campaigns.

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

Configurables

Batch source

Section	User Configuration Label	Label Description	Options	Default	Variable	User Widget
Basic	Authentication type	The way, how user would like to be authenticated to the SendGrid acoount	Basic Api Key	Basic	auth-type	radio-group
	API Key	The SendGrid API Key taken from the SendGrid account of choice	Shows if "Authentication type" is set to "api"		sendgrid-api-key	securekey-text
	Username	Login name for SendGrid	Shows if "Authentication type" is set to "basic"		username	textbox
	Password	Password for the SendGrid account	Shows if "Authentication type" is set to "basic"		password	password
	Data Source Types	List of data source groups	Marketing Campaigns Fields Stats Fields Suppressions Fields		dataSourceTypes	multi-select
	Marketing Campaign Objects	One of the above sources picked from list	Marketing Campaigns Fields Automation Single Sends Senders Contacts Segments

...

dataSourceMarketing

multi-select

Statistic Objects

One of the above sources picked from list

Stats

Global Stats
Category Stats
Advanced Stats

dataSourceStats

multi-select

Suppression Objects

One of the above sources picked from list

Suppressions

Bounces
Global Unsubscribes
Group Unsubscribes

...

* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks

...

		dataSourceSuppressions	multi-select
	Data Source Fields	List of fields to be retrieved	see "Data sources description" section below		dataSourceFields	multi-select

Options	Start Date	(required by some SG API Endpoints) The date in format YYYY-MM-DD, starting from which the data is requested.	Displayed depends on list of the selected data sources	not set	start-date	textbox
	End Date	(required by some SG API Endpoints) The date in format YYYY-MM-DD, the end date for the requested data	Displayed depends on list of the selected data sources	not set	end-date	textbox
Error Handling	On Record Error	How to handle error in record processing	Skip error Send to error port Fail pipeline	Skip error	on-error	Radio Button (layout: block)

Batch sink

Section	User Configuration Label	Label Description	Options	Default	Variable	User Widget

Standard

Basic

Authentication type

The way, how user would like to be authenticated

to the SendGrid account

Possible options:

Basic
Api Key

Required field

Basic

auth-type

radio-group

API Key

The

SendGrid

SendGrid API Key

taken

taken from the SendGrid account

of choice

	Shows if "Authentication type" is set to "api"		sendgrid-api-key

Text Box

securekey-text
	Username	Login name for

Sendgrid

the SendGrid account	Shows if "Authentication type" is set to "basic"		username

Text Box

textbox
	Password

Login password for the username specified above

Password for the SendGrid account	Shows if "Authentication type" is set to "basic"		password

Text Box

password

Data Source

	From

One of the above sources picked from listdata-sourceText BoxData Source FieldsList of fields to be retrieveddata-source-fieldsText BoxError HandlingOn Record ErrorHow to handle error in record processing

Skip error
Send to error port
Fail pipeline

Skip erroron-errorRadio Button (layout: block)

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

The author of the message	Required field		from	textbox
	Recipient address source	Recipients addresses source selection	The recipient address possible sources: Input Record (input) Configuration (config) Would be displayed depends on selection: recipientColumnName recipientConfigAddressList Required field		recipientAddressSource	radio-group
	Column name for recipients addresses	Name of the column with coma-separated list of recipients			recipientColumnName	textbox
	Recipient address list	List of mail recipients			recipientConfigAddressList	csv
	Email subject	Email Subject	Required field		mailSubject	textbox
	Email body column name	Name of the column for the mail content	Required field		bodyColumnName	textbox
	Reply To	Email address to which the author of the message suggests that replies be sent			replyTo	textbox
Options	Custom Mail Footer	Footer feature setting switcher	Possible states: On (true) Off (false)	Off	footerEnabled	toggle
	Footer HTML	The default footer which would be included to every email			footerHtml	textarea
	Sandbox Mode	Allows to send a test email to ensure that your request body is valid and formatted correctly	Possible states: On (true) Off (false)	Off	sandboxMode	toggle
	Click Tracking	Allows to track whether a recipient clicked a link in a email	Possible states: On (true) Off (false)	Off	clickTracking	toggle
	Open Tracking	Allows to track whether the email was opened or not, by including a single pixel image in the body of the content. When the pixel is loaded, SendGrid can log that the email was opened	Possible states: On (true) Off (false)	Off	openTracking	toggle
	Subscription Tracking	Allows to insert a subscription management link at the bottom of the text and html bodies of an email	Possible states: On (true) Off (false)	Off	subscriptionTracking	toggle

Design Tips and references

SendGrid v3 API:

- https://sendgrid.api-docs.io/v3.0/

- https://sendgrid.com/docs/API_Reference/Web_APIapi_v3/index.html

SendGrid supports API v2 and v3 with Legacy Marketing Campaigns (LMC) API and New Marketing Campaigns(NMC) legacy and and SendGrid v3 API . As v2 and LMC API v3 legacy are outdated, plugin we going to support and use the latest SendGrid v3 API.

User Expectations

User will retrieve API Keys (Restricted Access) from their SendGrid account for Event Notification, Marketing Campaigns, Suppressions and Tracking.
User will enable Active Tracking in their account - Click, Open and Subscription tracking is enabled.
Uses SendGrid v3

...

REST API and Webhooks to sync your data.
Retrieves
- Marketing Campaigns
  - Automation
  - Single Sends
  - Senders
  - Contacts
  - Segments
  - Schedule *
- Stats
  - Global Stats
  - Category Stats
  - Advanced Stats
- Suppressions
  - Bounces
  - Global Unsubscribes
  - Group Unsubscribes
  - Spam Reports **
  - Blocks **
* is present as state in the list of Automation/Single Send, but could be parsed as separate list of both entities to represent the current schedule and be a dynamic entity
** currently no way to test or get info about those endpoints as for that, test account should have been reported for spam and have some blocks

Data source reference

Campaigns

Object	API Reference	API Premissions	Notes		Notes
Automations	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Single Sends	https://sendgrid.api-docs.io/v3.0/listssingle-sends/get-all-lists	*	Senderssingle-sends
Senders	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Contacts	https://sendgrid.api-docs.io/v3.0/senderscontacts/createget-a-sender-identity-1	v3 API allow to create sender identities, still looking for retrieving existing ones	Lists	all-contacts
Segments	https://sendgrid.api-docs.io/v3.0/listssegmenting-contacts/get-alllist-of-listssegments	*
RecipientsGlobalStats	https://sendgrid.api-docs.io/v3.0/contactsstats/getretrieve-global-allemail-contactsstatistics
Unsubscribe groupCategoryStats	https://sendgrid.api-docs.io/v3.0/suppressions-unsubscribe-groupscategories/retrieve-allemail-suppressionstatistics-groups-associated-with-the-user	Advanced Stats	for-categories
AdvancedStats	https://sendgrid.com/docs/API_Reference/Web_API_v3/Stats/advanced.html
Bounces	https://sendgrid.api-docs.io/v3.0/bounces-api/retrieve-all-bouncesCategory Stats
Global Unsubscribes	-	No documentation available, but the usage could be revers engineered through debugging of official UI
Group Unsubscribes	https://sendgrid.api-docs.io/v3.0/categoriessuppressions-unsubscribe-groups/retrieve-email-statistics-for-categoriesGlobal Stats-all-suppression-groups-associated-with-the-user
Spam Reports	https://sendgrid.api-docs.io/v3.0/statsspam-reports-api/retrieve-globalall-emailspam-statisticsreports
Marketing CampaignsBlocks	https://sendgrid.api-docs.io/v3.0/singleblocks-sendsapi/get-all-single-sends	Schedule	TODO	Segments	https://sendgrid.api-docs.io/v3.0/segmenting-contacts/get-list-of-segments retrieve-all-blocks

Data sources description

Field name	Type
Automation
id	string
name	string
status	string
type	string
message_count	number
created_at	string
updated_at	string
live_at	string
SingleSends
id	string
name	string
status	string
created_at	string
updated_at	string
is_abtest	boolean
Senders
id	string
nickname	string
address	string
address_2	string
city	string
country	string
state	string
zip	number
locked	boolean
created_at	string
updated_at	string
from_email	string
from_name	string
verified_status	string
verified_reason	string
reply_to_name	string
reply_to_email	string
Contacts
id	string
first_name	string
last_name	string
list_ids	list[string]
created_at	string
updated_at	string
email	string
Segments
id	string
name	string
parent_list_id	string
created_at	datetime
updated_at	datetime
sample_updated_at	datetime
contacts_count	number
GlobalStats
date	string
metrics_blocks	number
metrics_bounce_drops	number
metrics_bounces	number
metrics_clicks	number
metrics_deferred	number
metrics_invalid_emails	number
metrics_opens	number
metrics_processed	number
metrics_requests	number
metrics_spam_report_drops	number
metrics_spam_reports	number
metrics_unique_clicks	number
metrics_unique_opens	number
metrics_unsubscribe_drops	number
metrics_unsubscribes	number
CategoryStats
name	string
type	string
date	string
metrics_blocks	number
metrics_bounce_drops	number
metrics_bounces	number
metrics_clicks	number
metrics_deferred	number
metrics_delivered	number
metrics_invalid_emails	number
metrics_opens	number
metrics_processed	number
metrics_requests	number
metrics_spam_report_drops	number
metrics_spam_reports	number
metrics_unique_clicks	number
metrics_unique_opens	number
metrics_unsubscribe_drops	number
metrics_unsubscribes	number
AdvancedStats
name	string
type	string
date	string
metrics_clicks	number
metrics_opens	number
metrics_unique_clicks	number
metrics_unique_opens	number
Bounces
created	string
email	string
reason	string
status	string
GlobalUnsubscribes
created	string
email	string
GroupUnsubscribes
id	string
name	string
description	string
is_default	boolean
last_email_send_at	string
unsubscribes	number

Authorization

Plugin supporting Basic and API Key authorization, with using API Key by default when provided.

Rate limiting

SendGrid v3 API have an request rate limiting. The API requester have integrated way to retry over time to fetch wait for penalty time.

Approach

Create a module sendgrid plugin in sendgrid project with use of existing SendGrid library. Add SendGrid objects with their deserialization from API and transformation to structured records.

UI Impact or Changes

SendGrid source and sink are separate entries with SendGrid logo in source and sink lists.

Test Case(s)

TODO

Sample Pipeline

TODO

Versions Compared

Old Version 9

New Version Current

Key

Introduction

User

Storie(s)

Plugin Type

Configurables

Batch source

Batch sink

Plugin Type

Design Tips and references

User Expectations

Data source reference

Data sources description

Automation

SingleSends

Senders

Contacts

Segments

GlobalStats

CategoryStats

AdvancedStats

Bounces

GlobalUnsubscribes

GroupUnsubscribes

Authorization

Rate limiting

Approach

UI Impact or Changes

Test Case(s)

Sample Pipeline

Page Comparison

Versions Compared

Old Version 9

New Version Current

Key

<span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-0">[data-colorid=</span>

User

Storie(s)

Plugin Type

Configurables

Batch source

Batch sink

Plugin Type

Design Tips and references

User Expectations

Data source reference

Data sources description

Automation

SingleSends

Senders

Contacts

Segments

GlobalStats

CategoryStats

AdvancedStats

Bounces

GlobalUnsubscribes

GroupUnsubscribes

Authorization

Rate limiting

Approach

UI Impact or Changes

Test Case(s)

Sample Pipeline