Checklist
...
- User stories documented (Shankar)
- User stories reviewed (Nitin)
- Design documented (Shankar/Vinisha)
- Design reviewed (Terence/Andreas)
- Feature merged ()
- Examples and guides ()
- Integration tests ()
- Documentation for feature ()
- Blog post
Usecase
- User wants to group log messages at application level and write multiple separate log files for each application. Example
...
- :
application-dir/{audit.log, metrics.log, debug.log}
...
- User wants to write these log files to a configurable path in HDFS.
- User also wants to be able to configure rolling policy for these log files similar to log-back.
...
User Stories
For each application, user wants to collect the application's logs into multiple logs files based on log level
- For each application, user wants to configure a location in HDFS to be used to store the collected logs.
- For each application, User wants the application log files stored in text format.
...
Introduce Log Processor, FileWriter and RotationPolicy interfaces. Pluggable in CDAP Log-saver.
...
| Code Block |
|---|
public interface LogProcessor {
/**
* Called during initialize, passed properties for log processor.
*
* @param properties
*/
void initialize(Properties properties);
/**
* Process method will be called with iterator of log messages, log messages received will be in sorted order,
* sorted by timestamp. This method should not throw any exception,exceptions. ifIf any unchecked exceptions are thrown,
* log.saver will log an error and the processor will not receive messages.
* Will start receiving messages on log.saver startup.
*
* @param events list of {@link LogEvent}
*/
void process(Iterator<LogEvent> events);
/**
* stop logprocessor
*/
void destroy();
} |
| Code Block |
|---|
class LogEvent {
/**
* Logging event
**/
ILoggingEvent iLoggingEvent;
/**
* CDAP program entity-id
**/
EntityId entityId;
} |
Currently, we only have AvroFileWriter in Log.saver, ; we can create an interface for users to configure the FileWriter to provide if needed. This provides (would?) provide the option to abstract certain common logic for file rotation, maintaining created files, etc. in Log saver and a custom file writer can implement the other methods specific to it's its logic,
Example: Creating files in HDFS and maintaining the size of events processed is maintained by custom FileWriter extension.
| Code Block |
|---|
public interface FileWriter {
/**
* append events to file
**/
void append(Iterator<LogEvent> events);
/**
* create a file corresponding to the entityId and timestamp and return the file
**/
File createFile(EntityId entityId, long timestamp);
/**
* close the file
**/
void close(File file, long timestamp);
/**
* flush the contents
**/
void flush();
}
|
...