Introduction

Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud Platform. Like Firebase Realtime Database, it keeps your data in sync across client apps through realtime listeners and offers offline support for mobile and web so you can build responsive apps that work regardless of network latency or Internet connectivity. With CDAP, Cloud Firestore users should be able to ETL their Firestore data into an analytics warehouse, as well as load application data from various sources into Firestore.

Use case(s)

As an ETL developer, I would like to use application data stored in Firestore inside my data warehouse, so that I can use it for analytics purposes
As an application developer, I would like data from various sources to be loaded into Firestore, so I can power my app with data and insights in various systems.

User Storie(s)

As a user, I would like to create a pipeline using a Cloud Firestore source
As a novice user, I want to be able to retrieve data from Firestore without specifying a query
As an advanced user, I want to specify the query to retrieve data from Firestore
As a application developer, I want to use a Cloud Firestore sink in CDAP to load insights from various sources into Firestore
As an application developer, I want to add a document to Firestore using data from a relational database
As an application developer, I want to specify the mappings from my source data to documents inside Firestore
As an application developer, I want to specify the name of the document to create in Firestore
As an advanced user, I want to use a query to specify the mappings for documents in Firestore

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

Configurables

Batch Source

This section defines properties that are configurable for this plugin.

Section	User Facing Name	Type	Description	Constraints	Optional?	Default
Credentials	Project ID	Textbox				auto-detect
Credentials	Service Account	Textbox				auto-detect
Firestore properties	Mode	Toggle		Basic or Advanced		Basic
	Collection name	Select	Select the collection to retrieve. Only available wen Mode is Basic.	Can this be a select, or does it have to be a textbox
	Documents to pull	List	Specifies the list of documents to retrieve. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.		Yes	When both Fields to pull and Fields to skip are provided, throw a validation error
	Documents to skip	List	Specifies the list of documents to skip. All other fields are retrieved. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip.		Yes	When both Fields to pull and Fields to skip are provided, throw a validation error
	Query	Textarea	Allows user to enter a Firestore query to retrieve data. Only available when Mode is Advanced

NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire

Batch Sink

This section defines properties that are configurable for this plugin.

Section	User Facing Name	Type	Description	Constraints	Optional?	Default

Credentials	Project ID	Textbox				auto-detect
	Service Account	Textbox				auto-detect
	Database Id	Textbox	The Firestore database Id
Firestore properties
	Collection name	Select	Select the collection to write the data to	Can this be a select, or does it have to be a textbox
	Document Id Type	Select	Type of id assigned to documents written to the Cloud Firestore. The type can be one of two values: `Auto-generated id` - key will be generated by Cloud Firestore as a _Alpha Numeric ID_. `Custom name` - id will be provided as a field in the input records.	Select box		Auto-generated ID
	Document Id Alias	Textbox	The field that will be used as the document id when writing to Cloud Firestore. This must be provided when the Document Id Type is not auto generated		Yes
	Batch Size	Textbox	Maximum number of documents that can be passed in one batch to a Commit operation. The minimum value is `1` and maximum value is `500`		Yes

Design / Implementation Tips

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

Some future work – HYDRATOR-99999
Another future work – HYDRATOR-99999

Test Case(s)

Test case #1
Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data.

Pipeline #1

Pipeline #2

Table of Contents

Checklist

User stories documented
User stories reviewed
Design documented
Design reviewed
Feature merged
Examples and guides
Integration tests
Documentation for feature
Short video demonstrating the feature

Cloud Firestore plugins