WIP
Use Case
Users would like to have custom logic in filtering User wants to group log messages at application level and write the log messages to a log files in HDFS.multiple separate log files for each application. Example - application-dir/{audit.log, metrics.log, debug.log}. User wants to write these log files to a configurable path in HDFS. User also wants to be able to configure rolling policy for these log files similar to log-back.
Design
Introduce Log Processor, FileWriter and FileWriter RotationPolicy interfaces Pluggable in CDAP Logsaver.
...
Currently we only have AvroFileWriter in Log.saver, we can create an interface for users to configure the FileWriter to provide if needed. This provides the option to abstract certain common logic for file rotation, get previous maintaining created files, etc into an AbstractFileWriter in Log saver and custom file writer can implement the other methods specific to it's logic, example : writing to HDFS as text files, etc
Example : Creating files in HDFS and maintaining the size of events processed is maintained by custom FileWriter extension.
| Code Block |
|---|
public interface FileWriter {
/**
* append events to file
**/
void append(Iterator<ILoggingEvent>Iterator<LogEvent> events)
/**
* rotatecreate the olda file corresponding to the entityId and timestamp and get a new file
**/
File rotateFile(File file, EntityId entityId, long timestamp)
/**
* if file already exists for this entity and timestamp base, then the file would be returned, else new file would be createdreturn the file
**/
File getFilecreateFile(EntityId entityId, long timestamp)
/**
* close the file
**/
void close(File file, long timestamp)
/**
* close and delete the file
**/
void closeAndDelete(File file)
/**
* flush the contents
**/
void flush()
}
|
Step3: RotationPolicy
| Code Block |
|---|
public abstractinterface classRotationPolicy AbstractFileWriter{ implements FileWriter {/** public File rotateFile(File file, EntityId entityId, long timestamp) { // common-logic for rotating files } public getFile(EntityId entityId, long timestamp) { // common-logic for getting previously files } // etc..* For the logEvent, decide if we should rotate the log file corresponding to this event or not. */ boolean shouldRotateFile(LogEvent logEvent); } |
Option-1
Log Processor/File Writer Extensions run in the same container as log.saver.
...
- we can Stop the plugin (or)
- we can log an error and continue and stop the plugin after an error threshold.
4) FileWriterExtension will be used for file system operations (create, append, close) and RotationPolicyExtension will be used for deciding when to rotate the file.
5) stop the log processor when log.saver stops.
...
- as there can only be one logback.xml in a JVM and the logback is already configured for the log.saver container.
- logback doesn't existing implementation for writing to HDFS.
3) the configuration for logging location (base directory in hdfs) and logging class to use (SizeBasedRolling, etc) The properties required for extensions could be provided through cdap-site.xml for the extensions. These properties would be passed to the extension during initialize.
4) Log processor extension could provide an implementation of FileWriter interface (or extension of AbstractFileWriter) and RotationPolicy interfaces for HDFSFileWriter logic for the events it has processed using received from LogProcessor.
4) Future implementation for other policies have to be implemented at the end of extensions and can be configured through cdap-site.xml
...