Use Case
Checklist
- User stories documented (Albert/Vinisha)
- User stories reviewed (Nitin)
- Design documented (Albert/Vinisha)
- Design reviewed (Terence/Andreas)
- Feature merged ()
- Examples and guides ()
- Integration tests ()
- Documentation for feature ()
- Blog post
...
Usecase
User wants to group log messages at application level and write multiple separate log files for each application. Example - application-dir/{audit.log, metrics.log, debug.log}. User wants to write these log files to a configurable path in HDFS. User also wants to be able to configure rolling policy for these log files similar to log-back.
User Stories
...
Design
Introduce Log Processor, FileWriter and RotationPolicy interfaces Pluggable in CDAP Logsaver.
Programmatic API
...
Step1: LogProcessor
| Code Block |
|---|
public interface LogProcessor {
/**
* Called during initialize, passed properties for log processor
* @param properties
*/
void initialize(Properties properties);
/**
* Process method will be called with iterator of log messages, log messages received will be in sorted order,
* sorted by timestamp. This method should not throw any exception, if any unchecked exceptions are thrown,
* log.saver will log an error and the processor will not receive messages.
* Will start receiving messages on log.saver startup
*
* @param events list of {@link LogEvent}
*/
void process(Iterator<LogEvent> events);
/**
* stop logprocessor
*/
void destroy();
} |
| Code Block |
|---|
class LogEvent {
/**
* Logging event
**/
ILoggingEvent iLoggingEvent;
/**
* CDAP program entity-id
**/
EntityId entityId;
} |
Step2: FileWriter
Currently we only have AvroFileWriter in Log.saver, we can create an interface for users to configure the FileWriter to provide if needed. This provides the option to abstract certain common logic for file rotation, maintaining created files, etc in Log saver and custom file writer can implement the other methods specific to it's logic,
...
| Code Block |
|---|
public interface FileWriter {
/**
* append events to file
**/
void append(Iterator<LogEvent> events);
/**
* create a file corresponding to the entityId and timestamp and return the file
**/
File createFile(EntityId entityId, long timestamp);
/**
* close the file
**/
void close(File file, long timestamp);
/**
* flush the contents
**/
void flush();
}
|
Step3: RotationPolicy
| Code Block |
|---|
public interface RotationPolicy {
/**
* For the logEvent, decide if we should rotate the log file corresponding to this event or not.
*/
boolean shouldRotateFile(LogEvent logEvent);
} |
Approach
Option-1
Log Processor/File Writer Extensions run in the same container as log.saver.
...
1) As number of extensions increases and if a processor extension is slow, this could cause performance of log.saver to drop, which will affect the CDAP log.saver performance
Option-2 (or)
...
Improvement
Configure and Run a separate container for every log.processor plugin.
...