Introduction

Data loading in Snowflake (https://docs.snowflake.net/manuals/user-guide/data-load-overview.html) is performed in two, separate steps. The first step is to load the files into a staging area, and the second step is to trigger the upload with a COPY INTO step. This plugin will allow files to be written to the staging area for further processingaction plugin will execute both steps using Snowflake JDBC.

Use case(s)

• As a financial services company, I would like to build an analytics pipeline to push data into the Snowflake. I would leverage Hydrator, and use this plugin as a sink action to create files in the staging area . Finally, there would be an action plugin to and trigger the loading of the data.

User Storie(s)

• I would like to load processed data to the correct Snowflake staging area in the proper formatcreate files in the staging area and trigger loading of the data.
• I would like to have capability to load files from internal and external(s3) location.

Plugin Type

•  Batch Sink 
•  Real-time Sink
• Action

Configurables

This section defines properties that are configurable for this plugin. 

Design / Implementation Tips

• See this document for an overview: https://docs.snowflake.net/manuals/user-guide/data-load-overview.html

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• Test case #1
• Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2