Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform. You can use Cloud SQL with MySQL, PostgreSQL, or SQL Server (currently in beta). Cloud SQL plugins will allow CDAP users to read and write from/to their Cloud SQL instances without any technical knowledge.
Use case(s)
As an ETL developer, I would like to read my data in Cloud SQL, so that I can transform it using CDAP
As an ETL developer, I want to write the output of my pipeline to Cloud SQL, so that I can use the insights generated from my analytical processes to power my production database in Cloud SQL
User Storie(s)
As a user, I would like to create a pipeline using a Cloud SQL source
As a user, I want to create a pipeline using a Cloud SQL sink
As a user, I want to only specify a query, project ID and instance name to connect to Cloud SQL, so that I don't have to remember complex JDBC connection string syntax
As a user, I want to execute a SQL query on Cloud SQL as part of the control flow in my pipeline
As a user, I want to execute a SQL query on Cloud SQL as a notification of my pipeline's completion
As a user, I want to create a multi-table source and sink for Cloud SQL so that I can read multiple tables at the same time
As a user, I want to connect to Cloud SQL over a proxy.
Plugin Type
Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute
Configurables
Batch Source
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Project ID
Textbox
auto-detect
Service Account
Textbox
auto-detect
Database username
Textbox
The username to use to connect to the CloudSQL database
Database password
Password
The password to use to connect to the CloudSQL database
Cloud SQL properties
Database type
Radio button
MySQL or Postgres
N
Instance name
Select
Select the Cloud SQL instance name
Can this be a select, or does it have to be a textbox
N
Import Query
Textarea
The Query that specifies the data to pull from CloudSQL
N
Advanced
Bounding query
Textarea
The query to use to derive the bounds (min and max) to use to generate the splits
Y
Split Column
Select
The column to split by
Y
Number of splits
Number
The number of splits to generate
Y
1
NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire
Batch Sink
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Project ID
Textbox
auto-detect
Service Account
Textbox
auto-detect
Database Id
Textbox
The Firestore database Id
Firestore properties
Collection name
Select
Select the collection to write the data to
Can this be a select, or does it have to be a textbox
Document name
Select
Specifies the document to load the data to. Should be shown only when a collection has been selected
Can this be a select, or does it have to be a textbox