Introduction

Zuora is a subscription billing and finance solution. It is designed to automate recurring billing, collections, quoting, revenue recognition, and subscription metrics.

User Storie(s)

As a pipeline developer, I would like to pull data from an object in Zuora in a streaming pipeline, so that I can transform, enrich and synthesize it in a data pipeline.
As a pipeline developer, I would like to post data to an object in Zuora in a batch or streaming pipeline, so that I can update Zuora with my enriched data.
As a pipeline developer, I would like to pull data from an object in Zuora in a batch pipeline, so that I can transform, enrich and synthesize it in a data pipeline.
As a pipeline developer, for both batch and streaming sources, I would like the schema to be auto-populated based on the selected objects.
As a pipeline developer, for all interactions with Zuora, I would like Zuora API limits to be handled by the framework, so that I do not have to worry about them. (https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/Policies/Concurrent_Request_Limits)

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

Configurables

Batch source - single object/multi object

Should automatically populate schema for the selected object

Section	User Facing Name	Type	Description	Optional?	Default
Authentication	Authentication Type	radio button	The Authentication type to use, either OAuth2 or Basic. Defaults to Basic.	No	Basic
	Username	String	The Zuora ID to use to make request. Only required when authentication type is Basic.	Yes
	Password	password	The Zuora password. Only required when authentication type is Basic.	Yes
	Client Id	String	The Client ID of the OAuth2 client. Only required when Authentication Type is OAuth2	Yes
	Client Secret	password	The Client Secret that was displayed when the OAuth2 client was created. Only required when Authentication Type is OAuth2	Yes
	REST API Endpoint	String	The one of Zuora API endpoint to use: https://rest.zuora.com https://rest.eu.zuora.com https://rest.apisandbox.zuora.com https://rest.sandbox.eu.zuora.com https://rest.pt1.zuora.com	No
Basic	Object(s) to pull	select	The Zuora object to pull from the list at https://www.zuora.com/developer/api-reference/#section/Zuora-Object-Model	No	Account

Streaming source

Should automatically populate schema for the selected object

Section	User Facing Name	Type	Description	Optional?	Default
Authentication	Authentication Type	radio button	The Authentication type to use, either OAuth2 or Password or Authorization Cookie. Defaults to OAuth2.	No	OAuth2
	Access Key ID	String	The Zuora ID to use to make request. Only required when authentication type is password.	Yes
	Secret Key	password	The Zuora password. Only required when authentication type is password.	Yes
	Client Id	String	The Client ID of the OAuth2 client. Only required when Authentication Type is OAuth2	Yes
	Client Secret	password	The Client Secret that was displayed when the OAuth2 client was created. Only required when Authentication Type is OAuth2	Yes
	Authorization Cookie	password	The authorization cookie to use for session authentication. Only required when authentication type is Authorization Cookie.	Yes
Basic	Object to pull	select	The Zuora object to pull from the list at https://www.zuora.com/developer/api-reference/#section/Zuora-Object-Model	No	Account

Batch sink

Writes data to the selected object in Zuora

User Facing Name	Type	Description	Optional?	Default
Authentication Type	radio button	The Authentication type to use, either OAuth2 or Password or Authorization Cookie. Defaults to OAuth2.	No	OAuth2
Access Key ID	String	The Zuora ID to use to make request. Only required when authentication type is password.	Yes
Secret Key	password	The Zuora password. Only required when authentication type is password.	Yes
Client Id	String	The Client ID of the OAuth2 client. Only required when Authentication Type is OAuth2	Yes
Client Secret	password	The Client Secret that was displayed when the OAuth2 client was created. Only required when Authentication Type is OAuth2	Yes
Authorization Cookie	password	The authorization cookie to use for session authentication. Only required when authentication type is Authorization Cookie.	Yes
Object to write to	select	The Zuora objects to pull from the list at https://www.zuora.com/developer/api-reference/#section/Zuora-Object-Model	No	Account

Design / Implementation Tips

As the reference were used Zuora API version 230.0 of swagger definition. The definition could be obtained from the link in the article or be downloaded directly.

zuora/src location contains README.md file with addition information about auto-generated Zuora object definitions from the swagger file

API documentation: https://www.zuora.com/developer/api-reference/

Authorization

Plugin supporting Basic and OAuth2 authorization. OAuth2 method require user to enable such functionality in Zuora configuration or follow the instruction in the documentation:

https://knowledgecenter.zuora.com/CF_Users_and_Administrators/A_Administrator_Settings/Manage_Users; Section "Create an OAuth Client for a User"

Rate limiting

Zuora API have an request rate limiting for the object. The API requester have integrated way to retry over time to fetch wait for penalty time.

Documentation: https://knowledgecenter.zuora.com/BB_Introducing_Z_Business/Policies/Concurrent_Request_Limits

Approach(s)

Create a module zuora plugin in zuora project. Add Zuora objects with their deserialization from API and transformation to structured records.

Test Case(s)

TODO

Sample Pipeline