...

• Plugin type : Action
• Configurations
  • Query
    • SQL
      
      • Specifies the query to be executed. The query should be validated at deployment time through a dry-run of BigQuery job. 
    • Legacy SQL
      
      • Boolean option to specify whether the SQL specified above is a legacy sql
  • Execution Options
    • Model - Batch or Interactive
      • Specifies whether the SQL query should be batched and executed when resources are available or should run immediately. 
    • Cache Enabled/Disabled
      • Boolean option that specifies if cache can be used. Enabling caching helps boost query execution.
    • KMS Key Name
      • Specifies the name of the KMS key name to used for encrypting
    • Max Billing Tier
      • Specifies the billing tier to be used for executing the plugin
    • Non-Temporary Dataset
      • Specifies the name of the permanent dataset in which the results of execution should be stored. 
    • Non-Temporary Table
      • Specifies the name of the permanent table within a dataset in which the results of the execution should be stored. 
  • Project (Optional if running in GCP)
    • Project Id
      • Specifies the GCP project id. This is specified when CDAP is running outside of GCP environment. In GCP environment this is defaulted to 'auto-detect'
    • Service Account File
      • Specifies the path to service account file that is used to access Google BigQuery when run in a CDAP instance that is running outside of GCP. In GCP environment this is defaulted to 'auto-detect'

Example

Image Removed

A sample pipeline that reads data from a URL, parses, cleanse and writes to BQ staging tables before ELT'ing into separate tables. 

Image Added 

View file
name Health-Record-BQELT_v8-cdap-data-pipeline.json height

...

150