This document provides a step-by-step guide to build a Data Fusion data pipeline that reads data from Postgres, transforms the data, and writes to Cloud BigQuery.
Prerequisites
Before creating a Cloud Data Fusion pipeline that reads data from PostgreSQL and writes to BigQuery, make sure PostgreSQL is set up and accessible from the Cloud Data Fusion instance.
Instructions
Add a PostgreSQL password as a secure key to encrypt, and store it on a Data Fusion instance.
On any of the pipeline pages on Cloud Data Fusion, click the System Admin tab in the top right menu.
...
Replace “<your_password>” with your PostgreSQL password.
...
Connect to PostgreSQL using Wrangler
1. Navigate to the Wrangler page.
...
| Tip |
|---|
Once you’ve completed all the steps, you will be able to click on the newly-connected database in the left navigation panel and see the list of tables for that database. |
Transform data using Wrangler and build your Data Fusion pipeline
This section uses an example to demonstrate how to transform data. We search for a “persons” table and remove the “first_name” column from the table.
...
| Tip |
|---|
Once the above pipeline succeeds, preview the written data in BigQuery. |
Related articles
How to use Using JDBC drivers with Cloud Data Fusion
| Page Properties | ||
|---|---|---|
| ||
|