Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud Platform. Like Firebase Realtime Database, it keeps your data in sync across client apps through realtime listeners and offers offline support for mobile and web so you can build responsive apps that work regardless of network latency or Internet connectivity. With CDAP, Cloud Firestore users should be able to ETL their Firestore data into an analytics warehouse, as well as load application data from various sources into Firestore.
Use case(s)
- As an ETL developer, I would like to use application data stored in Firestore inside my data warehouse, so that I can use it for analytics purposes
- As an application developer, I would like data from various sources to be loaded into Firestore, so I can power my app with data and insights in various systems.
User Storie(s)
- As a user, I would like to create a pipeline using a Cloud Firestore source
- As a novice user, I want to be able to retrieve data from Firestore without specifying a query
- As an advanced user, I want to specify the query to retrieve data from Firestore
- As a application developer, I want to use a Cloud Firestore sink in CDAP to load insights from various sources into Firestore
- As an application developer, I want to add a document to Firestore using data from a relational database
- As an application developer, I want to specify the mappings from my source data to documents inside Firestore
- As an application developer, I want to specify the name of the document to create in Firestore
- As an advanced user, I want to use a query to specify the mappings for documents in Firestore
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Batch Source
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | |||||||
| Service Account | Textbox | auto-detect | Database Id | TextboxThe | ||||||
| Firestore | database IdFirestoreproperties | Mode | Toggle | Basic or Advanced | Basic | |||||
| Collection name | Select | Select the collection to retrieve. Only available wen Mode is Basic. | Can this be a select, or does it have to be a textbox | Document name | Select | Select the document to read from. Only available when Mode is Basic, and Collection name has been specified. Retrieves the entire document. | Should throw validation error when collection is specified and document is not | Yes | ||
| Fields to pull | Multi-selectDocuments to pull | List | Specifies the list of fields documents to retrieve. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip. | Yes | When both Fields to pull and Fields to skip are provided, throw a validation error | |||||
| Fields Documents to skip | Multi-selectList | Specifies the list of fields documents to skip. All other fields are retrieved. Only available when Mode is Basic. Users should either specify Fields to pull or Fields to skip. | Yes | When both Fields to pull and Fields to skip are provided, throw a validation error | ||||||
| Query | Textarea | Allows user to enter a Firestore query to retrieve data. Only available when Mode is Advanced |
NOTE: Upon selecting either the Collection name or the Query, users should see a button to generate the schema. This should populate the entire
Batch Sink
This section defines properties that are configurable for this plugin.
| Section | User Facing Name | Type | Description | Constraints | Optional? | Default | |
|---|---|---|---|---|---|---|---|
| Credentials | Project ID | Textbox | auto-detect | ||||
| Service Account | Textbox | auto-detect | |||||
| Database Id | Textbox | The Firestore database Id | |||||
| Firestore properties | |||||||
| Collection name | Select | Select the collection to write the data to | Can this be a select, or does it have to be a textbox | ||||
| Document | nameId Type | Select | Specifies the document to load the data to. Should be shown only when a collection has been selected | Can this be a select, or does it have to be a textboxType of id assigned to documents written to the Cloud Firestore. | Select box | Auto-generated ID | |
| Document Id Alias | Textbox | The field that will be used as the document id when writing to Cloud Firestore. | Yes | ||||
| Batch Size | Textbox | Maximum number of documents that can be passed in one batch to a Commit operation. | Yes |
Design / Implementation Tips
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature