Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform. You can use Cloud SQL with MySQL, PostgreSQL, or SQL Server (currently in beta). Cloud SQL plugins will allow CDAP users to read and write from/to their Cloud SQL instances without any technical knowledge.
Use case(s)
As an ETL developer, I would like to read my data in Cloud SQL, so that I can transform it using CDAP
As an ETL developer, I want to write the output of my pipeline to Cloud SQL, so that I can use the insights generated from my analytical processes to power my production database in Cloud SQL
User Storie(s)
As a user, I would like to create a pipeline using a Cloud SQL source
As a user, I want to create a pipeline using a Cloud SQL sink
As a user, I want to only specify a query, project ID and instance name to connect to Cloud SQL, so that I don't have to remember complex JDBC connection string syntax
As a user, I want to execute a SQL query on Cloud SQL as part of the control flow in my pipeline
As a user, I want to execute a SQL query on Cloud SQL as a notification of my pipeline's completion
As a user, I want to create a multi-table source and sink for Cloud SQL so that I can read multiple tables at the same time
As a user, I want to connect to Cloud SQL over a proxy.
Plugin Type
Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute
Configurables
Batch Source
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Project ID
Textbox
auto-detect
Service Account
Textbox
auto-detect
Database Id
Textbox
The Firestore database Id
Firestore properties
Mode
Toggle
Basic or Advanced
Basic
Collection name
Select
Select the collection to retrieve. Only available wen Mode is Basic.
Can this be a select, or does it have to be a textbox
Document name
Select
Select the document to read from. Only available when Mode is Basic, and Collection name has been specified. Retrieves the entire document.
Should throw validation error when collection is specified and document is not
Yes
Fields to pull
Multi-select
Specifies the list of fields to retrieve. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.
Yes
When both Fields to pull and Fields to skip are provided, throw a validation error
Fields to skip
Multi-select
Specifies the list of fields to skip. All other fields are retrieved. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.
Yes
When both Fields to pull and Fields to skip are provided, throw a validation error
Query
Textarea
Allows user to enter a Firestore query to retrieve data. Only available when Mode is Advanced
NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire
Batch Sink
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Project ID
Textbox
auto-detect
Service Account
Textbox
auto-detect
Database Id
Textbox
The Firestore database Id
Firestore properties
Collection name
Select
Select the collection to write the data to
Can this be a select, or does it have to be a textbox
Document name
Select
Specifies the document to load the data to. Should be shown only when a collection has been selected
Can this be a select, or does it have to be a textbox