Introduction 

Spark plugin that trains and predicts the label data based on the Gradient Boosted Tree Classifier.

Use-case

User wants to predict if the flight will be delayed or not based on some features of airline data:

Label → delayed and not delayed - delayed if 1.0 and 0.0 otherwise
Features → {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest}

User Stories

1. User should be able to train the data.

2. User should be able to classify the test data using the model build while training.
   

3. User should be able to provide the list of columns(features) to use for training.

4. User should be able to provide the list of columns(features) to be used for prediction.

5. User should be able to provide the column to be used as prediction field while training/regression.

6. User should be able to specify the maximum depth of the Gradient Boosted tree.

7. User should be able to specify maximum number of classes.

8. User should be able to specify maximum number of iterations.

9. User should be able to provide the file set name to save the training model.

10. User should be able to provide the path of the file set.

Example

Following is a simple example showing how GD Tree Trainer and Classifier would work to predict if the flight will be delayed or not.

For each flight, we have the following information:  

The GD Tree Trainer will train the data based on some features, for example : {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest .

The label for the first and second rows will be set to 1.0 and 0.0(delayed column value).

Trainer will save the model in a fileSet, which will be used later for predicting the delayed value using classification.

Conditions

Design

GD Tree Trainer

Input Json Format

{
  "name": "GDTreeTrainer",
  "type": "sparksink",
  "properties": {
        "fileSetName": "gd-tree-model",
        "path": "/home/cdap",
        "featureFields": "dofM,dofW,scheduleDepTime,scheduledArrTime,carrier,elapsedTime,origin,dest",
        "labelField": "delayed",
        "maxClass": "2",
        "maxDepth": "9",
        "maxIteration": "3"
   }
}

 Plugin will take above inputs from user and trains the model based on "featureFields" and  "labelField" fields as features and label points respectively.

Properties:

• fileSetName: The name of the FileSet to save the model to.
• path: Path of the FileSet to save the model to.
• featureFields: A comma-separated sequence of field names to used as features for training.
• labelField: It should be the column name from input structure record containing the data to be treated as label for prediction.
• maxClass: The number of classes to be used in training model. It should be of type integer.
• maxDepth: Maximum depth of the tree. For example, depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Default is 10

The model generated from this plugin will further be used by Logistic Regression Classifier plugin to classify the input data.