Introduction

Google provides BigQuery for querying massive datasets by enabling super-fast SQL queries against append-only tables using the processing power of Googles's infrastructure. Users can move their data into BigQuery and let it to handle the hard work. 

Now CDAP provides the interface for users to handle their datasets in BigQuery. 

...

Users want to integrate CDAP with their already stored dataset in Google BigQuery. 

User Stories

  1. As a user, I would like to run arbitrary queries synchronously against my datasets in BigQuery and pull those records in BigQuery and pull those records in a hydrator pipeline.

  2. As a user, i would like to store data from a Hydrator pipeline into a table (dataset) in BigQuery. If the table doesn't exist, it should be created.

Requirements

Requirements

1. User should provide the correct project id which he has access to. 
2.   

1. User should specify the limit time for the querying. 

...

Following is a simple example showing how BigQuery Source would work.

A dataset already exist in Google BigQuery:121

...

output Output schema is as follows:

...

User run query agains dataset in BigQuery and pull the records:

...

Configuration is specified as follows

      ♦ project Id

...

         ♦ vernal-seasdf-123456

     ♦ query

        ♦  SELECT name, count FROM baby_names ORDER BY count DESC LIMIT 3

 output:

Out put is as follows

...

Poll Results:

Using jobId:

Polling Latest Results: