Introduction 

Spark plugin that trains and predicts the label data based on the Gradient Boosted Tree Classifier.

Use-case

User wants to predict if the flight will be delayed or not based on some features of airline data:

Label → delayed and not delayed - delayed if 1.0 and 0.0 otherwise
Features → {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest}

User Stories

1. User should be able to train the data.

2. User should be able to classify the test data using the model build while training.
   

3. User should be able to provide the list of columns(features) to use for training.

4. User should be able to provide the list of columns(features) to be used for prediction.

5. User should be able to provide the column to be used as prediction field while training/regression.

6. User should be able to specify the maximum depth of the Gradient Boosted tree.

7. User should be able to specify maximum number of classes.

8. User should be able to specify maximum number of iterations.

9. User should be able to provide the file set name to save the training model.

10. User should be able to provide the path of the file set.

Example

Following is a simple example showing how GD Tree Trainer and Classifier would work to predict if the flight will be delayed or not.

For each flight, we have the following information:  

The GD Tree Trainer will train the data based on some features, for example : {dayOfMonth, weekday, scheduledDepTime, scheduledArrTime, carrier, elapsedTime, origin, dest .

The label for the first and second rows will be set to 1.0 and 0.0(delayed column value).

Trainer will save the model in a fileSet, which will be used later for predicting the delayed value using classification.

Conditions

Design

GD Tree Trainer

Input Json Format

{
  "name": "GDTreeTrainer",
  "type": "sparksink",
  "properties": {
        "fileSetName": "gd-tree-model",
        "path": "/home/cdap",
        "featuresfeaturesToInclude": "dofM,dofW,scheduleDepTime,scheduledArrTime,carrier,elapsedTime,origin,dest",
        "predictionFieldlabelField": "delayed",
        "maxClass": "2",
        "maxDepth": "9",
        "maxIteration": "3"
   }
}

Plugin will take above inputs from user and trains the model based on "featureFields" and  "labelField" fields as features and label points respectively.

Properties:

• fileSetName: The name of the FileSet to save the model to.
• path: Path of the FileSet to save the model to.
• featuresToInclude: A comma-separated sequence of fields to use for training. Features to be used, must be of type: int, double, float, long.
• featuresToExclude: A comma-separated sequence of fields to be excluded when training.
• cardinalityMapping: Mapping of the feature to the cardinality of that feature; required for categorical features
• labelField: It should be the column name from input structure record containing the data to be treated as label for prediction.
• maxClass: The number of classes to be used in training model. It should be of type integer.
• maxDepth: Maximum depth of the tree. For example, depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Default is 10

• maxIteration: The number of trees in the model. Each iteration produces one tree. Increased iteration value improves training data accuracy.

The model generated from this plugin will further be used by GD-Tree Classifier plugin to classify the input data.

GD Tree Classifier

Input Json Format

{
  "name": "GDTreeClassifier",
  "type": "sparkcompute",
  "properties": {
        "fileSetName": "gd-tree-model",
        "path": "/home/cdap",
        "featuresToInclude": "dofM,dofW,scheduleDepTime,scheduledArrTime,carrier,elapsedTime,origin,dest",
        "predictionField": "delayed"
   }
}

Classifier plugin will take above inputs from user and the GD-Tree model from the "fileSetName" and predict the flight data whether the flight will be delayed or not.

Properties:

• fileSetName: The name of the FileSet model.
• path: Path of the FileSet from which model needs to be retrieved.
• featuresToInclude: A comma-separated sequence of fields to use for classification. Features to be used, must be of type: int, double, float, long.
• featuresToExclude: A comma-separated sequence of fields to be excluded while classification.
• predictionField: It should be the column name in which the prediction data needs to be saved.

Both *featuresToInclude* and *featuresToExclude* fields cannot be specified simultaneously.
If inputs for *featuresToInclude* and *featuresToExclude* has not been provided then all the fields except label/prediction field will be used as feature fields.