Bigtable Sink
- Maksym Lozbin
Batch sink that writes data to Google Cloud Platforms's Bigtable.
User Expectations
- User specifies how they would like to handle errors during ingesting, depending on option chosen, the errors in processing are handled.
- User should be able to specify account credentials in configuration.
User Configurations
| Section | User Configuration Label | Label Description | Mandatory | Macro-enabled | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|---|---|
| Standard | Reference Name | This will be used to uniquely identify this source for lineage, annotating metadata, etc | + | + | referenceName | Text Box | ||
| Table | Database table name | + | + | table | Text Box | |||
| Column Family | Column family to use for all inserted rows. | + | + | columnFamily | Text Box | |||
| Instance ID | Bigtable instance ID | + | + | instance | Text Box | |||
| Project ID | The ID of the project in Google Cloud If not specified, will be automatically read from the cluster environment | + | project | Text Box | ||||
| Service Account File Path | Path on the local file system of the service account key used for If the plugin is run on a Google Cloud Dataproc cluster, the service account key does not need to be provided and can be set to 'auto-detect'. When running on other clusters, the file must be present on every node in the cluster. See Google's documentation on Service account credentials for details. | + | serviceFilePath | Text Box | ||||
| Key Alias | Name of the field for row key. | + | __key__ | keyAlias | Text Box |
Bigtable Overview
Storage model
Cloud Bigtable stores data in massively scalable tables, each of which is a sorted key/value map. The table is composed of rows, each of which typically describes a single entity, and columns, which contain individual values for each row. Each row is indexed by a single row key, and columns that are related to one another are typically grouped together into a column family. Each column is identified by a combination of the column family and a column qualifier, which is a unique name within the column family.
Each row/column intersection can contain multiple cells, or versions, at different timestamps, providing a record of how the stored data has been altered over time. Cloud Bigtable tables are sparse; if a cell does not contain any data, it does not take up any space.
General concepts
Designing a Cloud Bigtable schema is very different than designing a schema for a relational database. As you design your Cloud Bigtable schema, keep the following concepts in mind:
- Each table has only one index, the row key. There are no secondary indices.
- Rows are sorted lexicographically by row key, from the lowest to the highest byte string. Row keys are sorted in big-endian, or network, byte order, the binary equivalent of alphabetical order.
- Columns are grouped by column family and sorted in lexicographic order within the column family.
- All operations are atomic at the row level. For example, if you update two rows in a table, it's possible that one row will be updated successfully and the other update will fail. Avoid schema designs that require atomicity across rows.
- Ideally, both reads and writes should be distributed evenly across the row space of the table.
- In general, keep all information for an entity in a single row. An entity that doesn't need atomic updates and reads can be split across multiple rows. Splitting across multiple rows is recommended if the entity data is large (hundreds of MB).
- Related entities should be stored in adjacent rows, which makes reads more efficient.
- Cloud Bigtable tables are sparse. Empty columns don't take up any space. As a result, it often makes sense to create a very large number of columns, even if most columns are empty in most rows.
Supported data types
Cloud Bigtable treats all data as raw byte strings for most purposes. The only time Cloud Bigtable tries to determine the type is for increment operations, where the target must be a 64-bit integer encoded as an 8-byte big-endian value.
Implementation Details
- Task will be split using org.apache.hadoop.hbase.mapreduce.TableOutputFormat.
- Values will be converted into bytes using input schema.
- Supported input field types: boolean, int, long, float, double, bytes, string.
- All information about logical types will be lost when inserted because of schema-less DB nature.