Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Introduction

Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational databases on Google Cloud Platform. You can use Cloud SQL with MySQL, PostgreSQL, or SQL Server (currently in beta). Cloud SQL plugins will allow CDAP users to read and write from/to their Cloud SQL instances without any technical knowledge.

Use case(s)

  • As an ETL developer, I would like to read my data in Cloud SQL, so that I can transform it using CDAP
  • As an ETL developer, I want to write the output of my pipeline to Cloud SQL, so that I can use the insights generated from my analytical processes to power my production database in Cloud SQL

User Storie(s)

  • As a user, I would like to create a pipeline using a Cloud SQL source
  • As a user, I want to create a pipeline using a Cloud SQL sink
  • As a user, I want to only specify a query, project ID and instance name to connect to Cloud SQL, so that I don't have to remember complex JDBC connection string syntax
  • As a user, I want to execute a SQL query on Cloud SQL as part of the control flow in my pipeline
  • As a user, I want to execute a SQL query on Cloud SQL as a notification of my pipeline's completion
  • As a user, I want to create a multi-table source and sink for Cloud SQL so that I can read multiple tables at the same time
  • As a user, I want to connect to Cloud SQL over a proxy.

Plugin Type

  • Batch Source
  • Batch Sink 
  • Real-time Source
  • Real-time Sink
  • Action
  • Post-Run Action
  • Aggregate
  • Join
  • Spark Model
  • Spark Compute

Configurables

Batch Source

This section defines properties that are configurable for this plugin. 

SectionUser Facing NameTypeDescriptionConstraintsOptional?Default
CredentialsProject IDTextbox


auto-detect
Service AccountTextbox


auto-detect
Database IdTextboxThe Firestore database Id


Firestore propertiesModeToggle
Basic or Advanced
Basic
Collection nameSelectSelect the collection to retrieve. Only available wen Mode is Basic.Can this be a select, or does it have to be a textbox

Document nameSelectSelect the document to read from. Only available when Mode is Basic, and Collection name has been specified. Retrieves the entire document.Should throw validation error when collection is specified and document is notYes
Fields to pullMulti-selectSpecifies the list of fields to retrieve. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.
YesWhen both Fields to pull and Fields to skip are provided, throw a validation error
Fields to skipMulti-selectSpecifies the list of fields to skip. All other fields are retrieved. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.
YesWhen both Fields to pull and Fields to skip are provided, throw a validation error
QueryTextareaAllows user to enter a Firestore query to retrieve data. Only available when Mode is Advanced


NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire 

Batch Sink

This section defines properties that are configurable for this plugin. 


SectionUser Facing NameTypeDescriptionConstraintsOptional?Default
CredentialsProject IDTextbox


auto-detect
Service AccountTextbox


auto-detect
Database IdTextboxThe Firestore database Id


Firestore properties
Collection nameSelectSelect the collection to write the data toCan this be a select, or does it have to be a textbox


Document nameSelectSpecifies the document to load the data to. Should be shown only when a collection has been selectedCan this be a select, or does it have to be a textboxYes

Design / Implementation Tips

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

  • Some future work – HYDRATOR-99999
  • Another future work – HYDRATOR-99999

Test Case(s)

  • Test case #1
  • Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2



Table of Contents

Checklist

  • User stories documented 
  • User stories reviewed 
  • Design documented 
  • Design reviewed 
  • Feature merged 
  • Examples and guides 
  • Integration tests 
  • Documentation for feature 
  • Short video demonstrating the feature
  • No labels