Introduction

Use-Case

• User in their Hydrator Pipeline can receive the data from the source and would want to emit the tokenized data into a output source for one of the columns from the source using the specified delimiterTo analyze and train the dataset,user wants to break down the larger piece of text into individual words/tokens which can be further used to create feature lookup for prediction.Refer the Example section below.

User Stories

• As a Hydrator user,I want to tokenize the data in a column from source schema and output the tokens into output schema which will have a single column having tokenized data.
• As a Hydrator user I want to have configuration for specifying the column name from input schema on which tokenization has to be performed.
• As a Hydrator user I want to have configuration to specify the delimiter which could be used for tokenization.
• As a Hydrator user I want to have configuration to specify output column name wherein tokenized data will be emitted.

Conditions

• Source field ,to be tokenized,can be of only string type.
• User can tokenize single column only from the source schema.
• Output schema will have a single column of type string array.
  
  

Example

Input source:

Tokenizer:

• • User wants to tokenize the sentence data using “/” as a delimiter
  • Mandatory inputs from user:
  • Column on which tokenization to be done:”sentence”
  • Delimiter for tokenization:”/”
  • Output column name for tokenized data:”words”
  • Tokenizer plugin will tokenize “sentence” data from input source and put tokenized data in “words” in output.

Output:

Design

This is a sparkcompute type of plugin and is meant to work with Spark only.

Properties:

• columnToBeTokenized :Column name on which tokenization is to be done
• delimiter:Delimiter for tokenization
• outputColumn:Output column name for tokenized data 

Input JSON:

{

        "name": "Tokenizer",

        "plugin": {

        "name": "Tokenizer",

        "type": "sparkcompute",

        "label": "Tokenizer",

        "properties": {

           " columnToBeTokenized": "sentence",

           " delimiter": "/",

           " outputColumn": "words",

         }

       }

     }