...
The proposal is to determine the number of splits according to the number of staged files that were created using COPY INTO <location> command. The number of resulting files can be controlled using MAX_FILE_SIZE Copy Options. The proposal is to add "Max Split Maximum Split Size" Source configuration option property which will use MAX_FILE_SIZE Copy Optioncopy option.
LIST command returns a list of files that have been staged.
...
Section | User Configuration Label | Label Description | Options | Default | Variable | User Widget |
|---|---|---|---|---|---|---|
| General | Label | Label for UI. | textbox | |||
| Reference Name | Uniquely identified name for lineage. | referenceName | textbox | |||
| Account Name | Full name of Snowflake account. | accountName | textbox | |||
| Database | Database name to connect to. | database | textbox | |||
| Import Query | Query for import data. | importQuery | textarea | |||
| Credentials | Username | User identity for connecting to the specified database. | username | textbox | ||
| Password | Password to use to connect to the specified database. | password | password | |||
| Key Pair Authentication | Key Pair Authentication Enabled | If true, plugin will perform Key Pair authentication. |
| False | keyPairEnabled | toggle |
| Key File Path | Path to the private key file. | path | textbox | |||
| OAuth2 | OAuth2 Enabled | If true, plugin will perform OAuth2 authentication. |
| False | oauth2Enabled | toggle |
| Auth URL | Endpoint for the authorization server used to retrieve the authorization code. | authUrl | textbox | |||
| Token URL | Endpoint for the resource server, which exchanges the authorization code for an access token. | tokenUrl | textbox | |||
| Client ID | Client identifier obtained during the Application registration process. | clientId | textbox | |||
| Client Secret | Client secret obtained during the Application registration process. | clientSecret | password | |||
| Scopes | Scope of the access request, which might have multiple space-separated values. | scopes | textbox | |||
| Refresh Token | Token used to receive accessToken, which is end product of OAuth2. | refreshToken | textbox | |||
| Advanced | Maximum Split Size | Maximum split size specified in bytes | maxSplitSize | number | ||
| Connection Arguments | A list of arbitrary string tag/value pairs as connection arguments. See: https://docs.snowflake.net/manuals/user-guide/jdbc-configure.html#jdbc-driver-connection-string | connectionArguments | keyvalue |
...
- The table above is similar to the Option1.1 one except of missing splitter-related properties. Please, refer to the Design section for the proposal of the splitter design.
...