Introduction

Use-Case

• Tokenize the data on the basis of delimiter
• Tokenized output will be an array of delimited tokens
• If you want to have sentence to be broken into tokens of words
  • Source Field name : e.g sentence(Type: String)
  • Target field name : e.g words(Type: String[])
    
    

User Stories

• User should be able to specify the column name on which tokenization is to be done.
• User should be able to specify the output column name.
• User should be able to specify the delimiter which will be used by Tokenizer.
  
  

Conditions

• Source field can be of only string type.
• User can tokenize single column only from the source schema.
• Output schema will have a single column of type string array.
• Tokenized data will be converted to lower case
  
  

Example

Input source:

Tokenizer:

• • User wants to tokenize the sentence data using “/” as a delimiter
  • Mandatory inputs from user:
  • Column on which tokenization to be done:”sentence”
  • Delimiter for tokenization:”/”
  • Output column name for tokenized data:”words”
  • Tokenizer plugin will tokenize “sentence” data from input source and put tokenized data in “words” in output.

Output:

Design

Properties:

• columnToBeTokenized :Column name on which tokenization is to be done
• delimiter:Delimiter for tokenization
• outputColumn:Output column name for tokenized data 

Input JSON:

{

        "name": "Tokenizer",

        "plugin": {

        "name": "Tokenizer",

        "type": "sparkcompute",

        "label": "Tokenizer",

        "properties": {

           " columnToBeTokenized": "sentence",

           " delimiter": "/",

           " outputColumn": "words",

         }

       }

     }