Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction 

Is a aggregator plugin that allows one to sort rows based on fields you specify and whether they should be sorted in ascending or descending order. 

Use-case

User is processing web access log and as part of his data pipeline user is aggregating response codes. User uses “GROUP BY” aggregation plugin in the pipeline to count for web responses in each response category. At the end of the file, user wants to sort it based on count before being written to the file. 

User Stories

  • User should have ability to specify single or composite field for sorting the records in the output
  • User should be able to specify field (basic type) from a nested structure for sorting the records
  • User should specify for each field how the records should be sorted
  • User can specify only basic types - String, Int, Long, Short, Float, Double, Byte as key, in case any other types are specified then error is thrown to notify the user

Example

Following is a simple example showing how Order By would work.

Input

First NameLast NameAgeZip Code
JoltieRoot2932826
HenryZilka6296789
BabyTrump1076563
DonaldTrump7034566
IvankaTrump3494306
BipashaBasu3967543
BabyIITrump1032816

Configuration is specified as follows
  • Input Schema
    • First Name, String
    • Last Name, String
    • Age, Int
    • Zip Code, Long
  • Sort by 
    • Last Name, Ascending
    • Age, Ascending
    • Zipcode, Descending
  • Output Schema
    • First Name, String
    • Last Name, String
    • Age, Int
    • Zip Code, Long

Output is as follows


First NameLast NameAgeZip Code
BipashaBasu3967543
JolieRoot2932826
BabyTrump1076563
BabyIITrump1032816
IvankaTrump3494306
DonaldTrump7076563
HenryZilka6296789

Implementation Tips

  • Investigate how ‘Group Comparator’ and ‘Sort Comparator’ work together and be used to achieve the functionality for this plugin.
  • Build a simple map-reduce program to show understand how the above functionality work — Implement Sort Comparator using StructuredRecord
  • If the above works, then Data Pipeline Application Template need to be modified to the set the sort class comparator and this shouldn’t affect the other plugins.

Design 

The order by plugin will use the Secondary Sort technique to sort the values (in ascending or descending order) passed to each reducer.

This plugin looks as below:

CompositeKeyWritable.java

Code Block
languagejava
/**
 * CustomWritable for the composite key.
 */
public class CompositeKey implements Writable, WritableComparable<CompositeKey> {

  private String structureRecordJSON;  //StructuredReocrd will be received as JSON string from the mapper.
  private String sortFieldsJSON;  //List of fields to be sorted will be received as JSON string from the mapper.
  
  /**
  *This comparator controls the sort order of the keys.
  */
  public int compareTo(CompositeKey other) {
      //Compare the structuredRecord objects parsed from the json string using JSON
  }
}

Table of Contents

Table of Contents
stylecircle

Checklist

  •  User stories documented 
  •  User stories reviewed 
  •  Design documented 
  •  Design reviewed 
  •  Feature merged 
  •  Examples and guides 
  •  Integration tests 
  •  Documentation for feature 
  •  Short video demonstrating the feature