WIP
Use Case
Users would like to have custom logic in filtering log messages at application level and write the log messages to a log files in HDFS.
Design
Introduce Log Processor interface Pluggable in CDAP Logsaver.
API Design
public interface LogProcessor {
/**
* Called during initialize, passed properties for log processor
* @param properties
*/
void initialize(Properties properties);
/**
* Process method will be called with iterator of log messages, log messages received will be in sorted order,
* sorted by timestamp. This method should not throw any exception, if any unchecked exceptions are thrown,
* log.saver will log an error and the processor will not receive messages.
* Will start receiving messages on log.saver startup
*
* @param events list of {@link LogEvent}
*/
void process(Iterator<LogEvent> events);
/**
* stop logprocessor
*/
void destroy();
}
class LogEvent {
ILoggingEvent iLoggingEvent;
EntityId entityId;
}
Option-1
Plugins run in the same container as log.saver.
Lifecycle
1) Log saver will load and initialize the log processor plugin.
2) As Log saver processes the messages, the log processor's process will also be called with logging events.
3) on plugin error, we will stop the plugin. optionally we can log an error and continue and stop the plugin after a threshold.
4) stop the plugin when log.saver stops.
Class-Loading Isolation
1) Should the log processor plugins have separate classloaders (or) can they share the same classloader and the log.saver.
Having isolation helps with plugins to depend on different libraries, but should we allow them ? Its not clear.
2) If we create separate Class loader - we need to expose the following
- cdap-watchdog-api
- cdap-proto
- hadoop api
- classes exposed by CDAP system class loader ?
How a Log Processor extension Implementation will work? :
1) Log Processor would want to process the ILoggingEvent, format it to a log message string and write it to a destination in HDFS.
2) However the configuration for this log.processor cannot be a logback.xml.
- as there can only be one logback.xml in a JVM and the logback is already configured for the log.saver container.
- logback doesn't existing implementation for writing to HDFS.
3) the configuration for logging location (base directory in hdfs) and logging class to use (SizeBasedRolling, etc) could be provided through cdap-site.xml for the extensions. These properties would be passed to the extension during initialize.
4) We have to extract logic performed in Log.saver for managing file size,etc and reuse them in extensions.
4) Future implementation for other policies have to be handled at the end of extensions and configured through cdap-site.xml
Pros
1) Leverages scalability of log.saver
2) Utilization of existing resources , logic and processing. Leveraging sorted messaging capability of log.saver for plugins.
Cons
1) As number of extensions increases and if a processor extension is slow, this could cause performance of log.saver to drop, which will affect the CDAP log.saver
Option - 2:
Configure and Run a separate container for every log.processor plugin.
Log.saver could have capability to launch system and user plugin containers. Scalability of these plugin containers could be managed separately.