Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform. You can use Cloud SQL with MySQL, PostgreSQL, or SQL Server (currently in beta). Cloud SQL plugins will allow CDAP users to read and write from/to their Cloud SQL instances without any technical knowledge.
Use case(s)
- As an ETL developer, I would like to read my data in Cloud SQL, so that I can transform it using CDAP
- As an ETL developer, I want to write the output of my pipeline to Cloud SQL, so that I can use the insights generated from my analytical processes to power my production database in Cloud SQL
User Storie(s)
- As a user, I would like to create a pipeline using a Cloud SQL source
- As a user, I want to create a pipeline using a Cloud SQL sink
- As a user, I want to only specify a query, project ID and instance name to connect to Cloud SQL, so that I don't have to remember complex JDBC connection string syntax
- As a user, I want to execute a SQL query on Cloud SQL as part of the control flow in my pipeline
- As a user, I want to execute a SQL query on Cloud SQL as a notification of my pipeline's completion
- As a user, I want to create a multi-table source and sink for Cloud SQL so that I can read multiple tables at the same time
- As a user, I want to connect to Cloud SQL over a proxy.
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Batch Source
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | |||
| Service Account | Textbox | auto-detect | ||||
| Database username | Textbox | The username to use to connect to the CloudSQL database | ||||
| Database password | Password | The password to use to connect to the CloudSQL database | ||||
| Cloud SQL properties | Database type | Radio button | MySQL or Postgres | N | ||
| Instance name | Select | Select the Cloud SQL instance name | Can this be a select, or does it have to be a textbox | N | ||
| Import Query | Textarea | The Query that specifies the data to pull from CloudSQL | N | |||
| Advanced | Bounding query | Textarea | The query to use to derive the bounds (min and max) to use to generate the splits | Y | ||
| Split Column | Select | The column to split by | Y | |||
| Number of splits | Number | The number of splits to generate | Y | 1 |
NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire
Batch Sink
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | |||
| Service Account | Textbox | auto-detect | ||||
| Database Id | Textbox | The Firestore database Id | ||||
| Firestore properties | ||||||
| Collection name | Select | Select the collection to write the data to | Can this be a select, or does it have to be a textbox | |||
| Document name | Select | Specifies the document to load the data to. Should be shown only when a collection has been selected | Can this be a select, or does it have to be a textbox | Yes |
Design / Implementation Tips
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature