...

1. For each application, user wants to  collect the application's logs into multiple logs files based on log level 

2. For each application, user wants to configure a location in HDFS to be used to store the collected logs. 
3. For each application, User wants the application log files stored in text format. 
4. For each application, User would wants to configure the RotationPolicy of log files. 
5. For each application, user wants to configure different layout for formatting log messages for the different log files generated.

Design

Introduce Log Processor, FileWriter and RotationPolicy interfaces. Pluggable in CDAP Log-saver. 

...

Currently, we only have AvroFileWriter in Log.saver; we can create an interface for users to configure the FileWriter if needed. This (would?) provide provides the option to abstract certain common logic for file rotation, maintaining created files, etc. in Log saver and a custom file writer can implement the other methods specific to its logic,

...

1) Should the log processor plugins have separate class-loaders (or) can they share the same ClassLoader as the log.saver system. 

     Having isolation helps with processor extensions to depend on different libraries, but should we allow them ? 

2

2)  If we use same Class-loader as log.saver, dependencies of extensions can be added to the classpath, and the classes available in log.saver system (hadoop, proto, ec) can be filtered out from the extension, so we use the classes provided by log.saver.

3) However if there are multiple log.processor extensions, say one for writing to s3 and another for writing to splunk, the classes from their dependencies could  potentially conflict with each other if we use the system class-loader ?

4) If we create separate Class loader for each extension to provide class loader isolation - we need to expose the following 

• cdap-watchdog-api
• cdap-proto
• hadoop
• logback-classic ( we need ILoggingEvent)
• should we expose more classes ? 
  
• What if user wants to write to a kafka server or third-party storage s3 on the log.processor logic? Having separate class loader will help in these scenarios.

Sample Custom Log Plugin Implementation 

1) Log Processor would want to process the ILoggingEvent, format it to a log message string (maybe using log-back layout classes) and write it to a destination.

...