Introduction

Use-Case

If you want to have sentence to be broken into tokens of words

• Source Field name : e.g sentence(Type: String)
• Target field name : e.g words(Type: String[])

Conditions

• Source field can be of only string type
• User can tokenize single column only from the source
• Output schema will have a single column of type string array
  
  

Options

Following are the mandatory inputs that will be provided to user to configure

• Column name on which tokenization to be done
• Delimiter for tokenization
• Output column name for tokenized data

Example

Input source:

Tokenizer:

User wants to tokenize the sentence data using “/” as a delimiter

Mandatory inputs from user:

Column on which tokenization to be done:”sentence”

Delimiter for tokenization:”/”

Output column name for tokenized data:”words”

Tokenizer plugin will tokenize “sentence” data from input source and put tokenized data in “words” in output.

Output:

Design

Input JSON:

{

        "name": "Tokenizer",

        "plugin": {

        "name": "Tokenizer",

        "type": "sparkcompute",

        "label": " Tokenizer ",

        "properties": {

           " columnToBeTokenized ": "sentence",

           " delimiter ": "/",

           " outputColumn": "words",

         }

       }

     }