Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

SectionFieldTypeDescription

Basic

Configuration

Input FileName


String

Full Name of File including path ( URI)

Compress FileBooleanTrue / False
Compression AlgorithmString ( List)Gzip / Snappy. Applicable only if above Compress File is set to true
Encrypt FileBooleanTrue
PGP Public Key PathStringLocation of PGP public key. Path to File
PGP Public Key Access UseridStringUserid to access the public key incase security is enabled

PGP Public Key Access

password

StringPassword to access the key file
OutFilePathString

Path to store the output file from sync. The output filename will follow the format of <InputfileName Suffix>.gz.pgp

The file path URI can contain filesystem , Hdfs, gcs - google file system or cloud store.

MoveInputBooleanTrue / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed.
MoveFilePathStringPath to move the input on successful processing of the file.



Usecase 2


Image Added



FileDecompressDecrypt Plugin

The plugin support decrypting  files using PGP Public key and decompress file. 


Plugin Properties

SectionFieldTypeDescription

Basic

Configuration

FileName

Path


String

Full Name of File including path ( URI)

Location of PGK Public KeyStringKey Access ( userid )StringPasswordString

Output

Schema

Decrypt FileName (TBD)StringRecord with FileName with full URIDecrypt File contents Content (TBD)

Path containing file name or directory of files.


Recursive ProcessingBooleanTrue / False

DeCompress FileBooleanTrue / False

DeCompression AlgorithmString ( List)Gzip / Snappy. Applicable only if above Compress File is set to true

DeEncrypt FileBooleanTrue

PGP Private Key PathStringLocation of PGP public key. Path to File

PGP Private Key Access UseridStringUserid to access the public key incase security is enabled


PGP Private Key Access

password

StringPassword to access the key file





MoveInputBooleanTrue / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed.

MoveFilePathStringPath to move the input on successful processing of the file.

Output

Schema

OutputStringEach Row from the file read.





Queries