Introduction

A batch source that ingests data from dynamodb into hydrator pipelines.

Use case(s)

• For an organization all the spam emails are being dumped to dynamodb table. As a data scientist I want to train my machine learning models in hydrator pipelines based on the data from the dynamodb tables.
  
  

User Storie(s)

• User should be able to provide the table name in DynamoDb.
• User should be able to provide the AWS endpoint url for DynamoDb instance.
• User should be able to provide the AWS region id for DynamoDb instance.
• User should be able to provide the throughput for DynamoDb instance. (Dynamo db charges are incurred based on throughput and user should be able to control the throughput)
• User should be able to provide the AWS access id.
• User should be able to provide the AWS access key.
  
  
  

Plugin Type

•  Batch Source
•  Batch Sink 
•  Real-time Source
•  Real-time Sink
•  Action
•  Post-Run Action
•  Aggregate
•  Join
•  Spark Model
•  Spark Compute

Configurables

This section defines properties that are configurable for this plugin. 

Design / Implementation Tips

• Please refer our Mongo db plugins
• Please refer https://github.com/awslabs/emr-dynamodb-connector

Design

We will provide dropdown with the list of supported regions to user, to select the region for AWS Dynamo DB to connect to.

Approach(s)

Properties

• endpointUrl: The hostname and port for AWS Dynamo DB instance to connect to, separated by a colon. For example, localhost:8000.

• regionId: The region for AWS Dynamo DB to connect to.

• accessKey: Access key for AWS Dynamo DB.

• secretAccessKey: Secret access key for AWS Dynamo DB.

• tableName: The table to read the data from.

• throughput: Intended throughput for AWS Dynamo DB.

• query: The query that will fetch the data from table.

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• Test case #1
• Test case #2

Sample Pipeline

Pipeline #1

Pipeline #2