Run plugin allows user to run any executable binary installed and available on all Hadoop nodes. The user code is capable of processing the input record and return the output record to be further processed downstream in the pipeline.
Use-Case
Often times, in enterprise there are existing tools or systems that exist and perform complex transformations of data. These tools are time tested and have been running in production for long time. As more and more processing is being moved to Hadoop, users would like to slowly transition to running on Hadoop. In this case, they would like to have the ability to run the tools as in or with minor modifications. They have the tool or binary installed on all Hadoop nodes and they would like the ability to pass the processing record into the tool and retrieve the results back into the pipeline.
User Stories
User should be able to specify the fully path to binary or just the binary
User should be able to specify the arguments for the binary to be executed
User should be provided a specification about how the record is passed to binary (need to be designed)
If binary executable doesn’t exist or not in path or not executable, user should be notified appropriately during runtime
User is able to see the errors in log if the executable writes the errors to STDERR
User executable is able to read the record from STDIN
User executable is able to write the record to STDOUT
User executable is able to write the error records to a different FILE descriptor
User should be able to specify the error dataset to which data returned from executable special FILE descriptor
User will make sure the binary and it’s dependencies are available on all machines of the cluster and no capability needs to be added to the plugin for marshaling the executable
Design
Types of binary executable that will be supported by plugin are:
a) .bat
b) .exe
c) .sh d) .jar And to select the executable type, dropdown will be provided to the user.
Run Plugin Properties:
binaryName : The name of the binary or the full path to the binary, to be executed.
binaryType : Type of the binary to be executed. For example, .jar or .bat or .sh or .exe. Default is .jar.
inputSource : Source to read the data as an input to the binary. For example, Hydrator Source stage or STDIN.
inputSourceFields : A space-separated sequence of the fields that will be used as input arguments for the binary. Please make sure that the sequence of the fields are proper.
otherStaticArguments : A space-separated sequence of the static arguments apart from the input arguments that will be passed to the binary as an input. Please make sure that the sequence of the arguments are proper.