A source Plugin in Hydrator is needed to efficiently export data from Netezza. Netezza includes strategies for exporting data including the use of transient external tables, and a command line tool NZSQL that can export data to a flat file.
Use-case
A Hydrator User would like to incorporate Netezza data into a pipeline using a source plugin that does not require a JDBC connection to perform the export from Netezza.
User Stories
As a Hydrator User I want a plugin that I can use to export data from Netezza so that I can use the data in my pipeline.
As a Hydrator User I want a Netezza plugin that exports data efficiently using existing Netezza tools.
As a Hydrator User I want the export data capability of the Netezza plugin to be based on a sql query that I issue.
User should be able to specify credentials.
Passwords should not be viewable in plain text from inside pipeline viewer or hydrator studio.
User should be able to specify Netezza Instance
User should be able to specify location of nzqsql Utility.
User should be able to specify type of output.
User should be able to specify location of output files.
User should know of connectivity errors, or malformed queries/output identifier.
Example
Example for how the plugin should work
Implementation Tips
NZSQL is a command line utility that can run remotely to allow exporting. The output format can be specified to a flat file with specified delimiter or an external table.
Command example to output to a flat file: nzsql -d {dbname} -u {username} -h {host} -c "select * from table_with_90_columns " -t -F -o output.csv
Command example to use External Table: CREATE EXTERNAL TABLE '/tmp/export.csv' USING (DELIM ',') AS
SELECT foo.x, bar.y, bar.dt FROM foo, bar WHERE foo.x = bar.x;