Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
A source Plugin in Hydrator is needed to efficiently export data from Netezza. Netezza includes strategies for exporting data including the use of transient external tables, and a command line tool NZSQL that can export data to a flat file.
Use-case
A Hydrator User would like to incorporate Netezza data into a pipeline using a source plugin that does not require a JDBC connection to perform the export from Netezza.
User Stories
- As a Hydrator User I want a plugin that I can use to export data from Netezza so that I can use the data in my pipeline.
- As a Hydrator User I want a Netezza plugin that works exports data efficiently using existing Netezza tools.
- As a Hydrator User I want the export data capability of the Netezza plugin to be based on a sql query that I issue.
Example
Example for how the plugin should work
Implementation Tips
NZSQL is a command line utility that can run remotely to allow exporting. The output format can be specified to a flat file with specified delimiter or an external table.
nzsql information: Netezza Docs
External Table: Netezza Docs
Command example to output to a flat file: nzsql -d {dbname} -u {username} -h {host} -c "select * from table_with_90_columns " -t -F -o output.csv
Command example to use External Table: CREATE EXTERNAL TABLE '/tmp/export.csv' USING (DELIM ',') AS
SELECT foo.x, bar.y, bar.dt FROM foo, bar WHERE foo.x = bar.x;
Design
Design:
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature