Introduction

A batch sink that pushes data from hydrator pipelines into dynamoDb tables.

Use case(s)

• An organization wants to parse the logs generated by a system and want to store the metadata in dynamodb tables.

User Storie(s)

• • User should be able to provide the table name in DynamoDb. 
  • User should be able to provide the primary key of the table. 
  • User should be able to provide the type of primary key (hash or range).
  • The table should be created if it is not already existing. 
  • User should be able to provide the AWS endpoint url for DynamoDb instance.
  • User should be able to provide the AWS region id for DynamoDb instance.
  • User should be able to provide the AWS access id.
  • User should be able to provide the AWS access key.

Plugin Type

• Batch Source
• Batch Sink 
• Real-time Source
• Real-time Sink
• Action
• Post-Run Action
• Aggregate
• Join
• Spark Model
• Spark Compute

Configurables

This section defines properties that are configurable for this plugin. 

Design / Implementation Tips

• • For Testing purposes tables can be created either using AWS cli or using java code http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/JavaDocumentAPITablesExample.html
  • AWS dynamoDb cli refrence http://docs.aws.amazon.com/cli/latest/reference/dynamodb/
  • Java example for CRUD operations http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/batch-operation-document-api-java.html
  • Please reuse/modify RecordReader, InputFormat classes present here https://github.com/awslabs/emr-dynamodb-connector

Design

Approach(s)

Properties

• endpointUrl: aws endpoint http://docs.aws.amazon.com/general/latest/gr/rande.html#ddb_region This could be reconstructed using regionId.

• regionId: The region for AWS Dynamo DB to connect to.

• accessKey: Access key for AWS Dynamo DB.

• secretAccessKey: Secret access key for AWS Dynamo DB.
  
  
• tableName: The table to read the data from.
  
  
• primaryKey: The field name to be used as priary key.

Security

Limitation(s)

Future Work

Test Case(s)

Sample Pipeline