Introduction

An option of IAM role based authentication in the existing S3 source and sink plugins.

Use case(s)

• In the S3 source and S3 sink(Avro and Parquet) plugins,there should be a provision for user to select authentication mechanism for S3.User should have an option to select IAM role based authentication in the plugins.

User Storie(s)

• As a pipeline user,i want to have an option of IAM role based authentication in the S3 source and sink plugins in Hydrator.
• As a pipeline user,i want access ID and access key to be mandatory for Access Credentials authentication method.
  
  

Plugin Type

• Batch Source
• Batch Sink 
• Real-time Source
• Real-time Sink
• Action
• Post-Run Action
• Aggregate
• Join
• Spark Model
• Spark Compute

Configurables

New Configuration would be added in the S3 plugin

Authentication method to access S3.

Defaults to Access Credentials

Design / Implementation Tips

• Tip #1
• Tip #2

Design

{
"widget-type": "select",
"label": "Authentication Method",
"name": "authenticationMethod",
"widget-attributes": {
"values": [
"Access Credentials",
"IAM"
],
"default": "Access Credentials"
}
}

Approach(s)

1.When user selected IAM role based authentication method,need to omit the properties related to keys.

2.S3 regions which are supporting both the signature versions(Version 2 and Version 4) are only supported.

3.User need to have AWS environment only to use IAM role based authentication.Non-EC2 environment can not be used.

3. User would have to use s3a hadoop client only to use IAM authentication.(URI scheme: s3a://)

Properties

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• S3batch source with IAM role based authentication
• S3batchsource with key credentials
• S3Avrosink with IAM role based authentication
• S3AvroSink with key credentials
• S3ParquetSink with IAM role based authentication
• S3ParquetSink with key credentials

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2