Checklist

•  User stories documented (Albert/VinishaShankar)
•  User stories reviewed (Nitin)
•  Design documented (AlbertShankar/Vinisha)
•  Design reviewed (Terence/Andreas)
•  Feature merged ()
•  Examples and guides ()
•  Integration tests () 
•  Documentation for feature ()
•  Blog post

Usecase

User wants to group log messages at application level and write multiple separate log files for each application. Example - application-dir/{audit.log, metrics.log, debug.log}. User wants to write these log files to a configurable path in HDFS. User also wants to be able to configure rolling policy for these log files similar to log-back. 

User Stories

1. For each application, user wants to  collect the application's logs into multiple logs files based on log level 

2. For each application, user wants to configure a location in HDFS to be used to store the collected logs. 
3. For each application, User wants the application log files stored in text format. 

Design

Introduce Log Processor, FileWriter and RotationPolicy interfaces Pluggable in CDAP LogsaverLog-saver. 

Programmatic API

public interface LogProcessor {

  /**
   * Called during initialize, passed properties for log processor
   * @param properties
   */
  void initialize(Properties properties);

  /**
   * Process method will be called with iterator of log messages, log messages received will be in sorted order,
   * sorted by timestamp. This method should not throw any exception, if any unchecked exceptions are thrown,
   * log.saver will log an error and the processor will not receive messages.
   * Will start receiving messages on log.saver startup
   * 
   * @param events list of {@link LogEvent}
   */
  void process(Iterator<LogEvent> events);

  /**
   * stop logprocessor
   */
  void destroy();
}

...

public interface RotationPolicy {
  /**
   * For the logEvent, decide if we should rotate the log file corresponding to this event or not.
   */
  boolean shouldRotateFile(LogEvent logEvent);
}

Approach

Option-1

Log Processor/File Writer Extensions run in the same container as log.saver. 

...

1) As number of extensions increases and if a processor extension is slow, this could cause performance of log.saver to drop, which will affect the CDAP log.saver performance

Option-2 (or) Improvement 

Configure and Run a separate container for every log.processor plugin. 

...