...
Pass the log level information as preferences and change the log level correspondingly once the program starts. With the property of the preferences, the log level can be set through namespace, application and program level. The key of the preferences should be “system.log.level”, and the value will be one of the valid log levels. (ALL, OFF, TRACE, DEBUG, INFO, WARN, ERROR) If the value is invalid log levels, this log level change will be ignored and the default log level “DEBUG” "INFO" will be used due to Level.toLevel() function.
In AbstractProgramTwillRunnable, DistributedMapReduceTaskContextProvider, and SparkRuntimeContextProvider, we get the log level information by the key “system.log.level” from ProgramOptions. Make LogAppenderInitializer’s initialize method accept the log level and change the root logger’s log level.
...
Extract the prefix “system.log.level”, get all the classes and levels. Then pass them to LogAppenderInitializer to change the log level for their loggers. For a flow, preferences with key format "flowlet.[flowletname].system.log.level.[class]" can be passed to change the log level of a specific flowlet. The idea is like how scoping works for system.resources.memory, we extract the scope of flowlet.name, and set the log level in FlowTwillRunnable.
Step 3: Remove some settings in logback-container.xml
...