Introduction 

Spark plugin that trains and predicts the label data based on the Gradient Boosted Tree Classifier.

Use-case

User wants to predict if the flight will be delayed or not based on some features of airline data:

Label → delayed and not delayed - delayed if 1.0 and 0.0 otherwise
Features → {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest}

User Stories

1. User should be able to train the data.

2. User should be able to classify the test data using the model build while training.
   

3. User should be able to provide the list of columns(features) to use for training.

4. User should be able to provide the list of columns(features) to be used for prediction.

5. User should be able to provide the column to be used as prediction field while training/regression.

6. User should be able to specify the maximum depth of the Gradient Boosted tree.

7. User should be able to specify maximum number of classes.

8. User should be able to specify maximum number of iterations.

9. User should be able to provide the file set name to save the training model.

10. User should be able to provide the path of the file set.

Example

Following is a simple example showing how GD Tree Trainer and Classifier would work to predict if the flight will be delayed or not.

For each flight, we have the following information:  

The GD Tree Trainer will train the data based on some features, for example : {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest .

The label for the first and second rows will be set to 1.0 and 0.0(delayed column value).

Trainer will save the model in a fileSet, which will be used later for predicting the delayed value using classification.

Conditions

Design

GD Tree Trainer

Input Json Format

{
  "name": "GDTreeTrainer",
  "type": "sparksink",
  "properties": {
        "fileSetName": "gd-tree-model",
        "path": "/home/cdap",
        "featureFields": "dofM,dofW,scheduleDepTime,scheduledArrTime,carrier,elapsedTime,origin,dest",
        "labelField": "delayed",
        "maxClass": "2",
        "maxDepth": "9",
        "maxIteration": "3"
   }
}