Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform. You can use Cloud SQL with MySQL, PostgreSQL, or SQL Server (currently in beta). Cloud SQL plugins will allow CDAP users to read and write from/to their Cloud SQL instances without any technical knowledge.
Use case(s)
- As an ETL developer, I would like to read my data in Cloud SQL, so that I can transform it using CDAP
- As an ETL developer, I want to write the output of my pipeline to Cloud SQL, so that I can use the insights generated from my analytical processes to power my production database in Cloud SQL
User Storie(s)
- As a user, I would like to create a pipeline using a Cloud SQL source
- As a user, I want to create a pipeline using a Cloud SQL sink
- As a user, I want to only specify a query, project ID and instance name to connect to Cloud SQL, so that I don't have to remember complex JDBC connection string syntax
- As a user, I want to execute a SQL query on Cloud SQL as part of the control flow in my pipeline
- As a user, I want to execute a SQL query on Cloud SQL as a notification of my pipeline's completion
- As a user, I want to create a multi-table source and sink for Cloud SQL so that I can read multiple tables at the same time
- As a user, I want to connect to Cloud SQL over a proxy.
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Batch Source
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | |||
| Service Account | Textbox | auto-detect | ||||
| Database Id | Textbox | The Firestore database Id | ||||
| Firestore properties | Mode | Toggle | Basic or Advanced | Basic | ||
| Collection name | Select | Select the collection to retrieve. Only available wen Mode is Basic. | Can this be a select, or does it have to be a textbox | |||
| Document name | Select | Select the document to read from. Only available when Mode is Basic, and Collection name has been specified. Retrieves the entire document. | Should throw validation error when collection is specified and document is not | Yes | ||
| Fields to pull | Multi-select | Specifies the list of fields to retrieve. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip. | Yes | When both Fields to pull and Fields to skip are provided, throw a validation error | ||
| Fields to skip | Multi-select | Specifies the list of fields to skip. All other fields are retrieved. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip. | Yes | When both Fields to pull and Fields to skip are provided, throw a validation error | ||
| Query | Textarea | Allows user to enter a Firestore query to retrieve data. Only available when Mode is Advanced |
NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire
Batch Sink
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | |||
| Service Account | Textbox | auto-detect | ||||
| Database Id | Textbox | The Firestore database Id | ||||
| Firestore properties | ||||||
| Collection name | Select | Select the collection to write the data to | Can this be a select, or does it have to be a textbox | |||
| Document name | Select | Specifies the document to load the data to. Should be shown only when a collection has been selected | Can this be a select, or does it have to be a textbox | Yes |
Design / Implementation Tips
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature