[logback-dev] [JIRA] (LOGBACK-1512) let SCHEDULED_EXECUTOR_POOL_SIZE be configurable

[QOS.CH (JIRA)]

Liang Xie created LOGBACK-1512:
----------------------------------

             Summary: let SCHEDULED_EXECUTOR_POOL_SIZE be configurable
                 Key: LOGBACK-1512
                 URL: https://jira.qos.ch/browse/LOGBACK-1512
             Project: logback
          Issue Type: Improvement
          Components: logback-classic, logback-core
    Affects Versions: 1.3.0-alpha5, 1.2.3, 1.1.11
         Environment: AWS m5.xlarge ec2 instance,    vCPU num: 4
            Reporter: Liang Xie
            Assignee: Logback dev list
         Attachments: cpu_spike.png, latency.png

Our log servers deployed logback component, the logback configuration contains tens of appenders,   this is one of the appender definition:
{quote}<appender name="rawdata"
 class="ch.qos.logback.core.rolling.RollingFileAppender">
 <file>/data/xxx/rawdata/rawdata.log</file>
 <encoder>
 <pattern>%m%n</pattern>
 </encoder>
 <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
 <fileNamePattern>/data/xxx/rawdata/rawdata.log.%d\{yyyy-MM-dd-HH}.gz</fileNamePattern>
 </rollingPolicy>
 </appender>
{quote}
 

please note the "fileNamePattern" value,   we use gzip compression,  it means in each hour beginning,  all the tens of files are renamed to tmp files,  then be compressed in the thread pool.   Currently, this thread pool core size is immutable, a fixed value: 8,  which came from [https://github.com/qos-ch/logback/commit/b946551439134] 

In an small ec2 env with tens of big log files(a few of our rolling raw log files size are 10 Gb), say 4 vcore,  the compression will be last several minutes with all vcores running in full speed.

It will disturb the normal user request latency espencially under a high throughput env.

Our proposed solution are as following:

1. let SCHEDULED_EXECUTOR_POOL_SIZE be configurable,   it is meaningful since the logback users have different cpu core configs,  different requirements(latency vs throughput)

2. let the compression level be configurable,  currently,  the logback uses JDK's GZIPOutputStream class, which deflate algorithm is DEFAULT_COMPRESSION always.  It would be great if we could make it be configurable.

 

We forked a branch,  after updating SCHEDULED_EXECUTOR_POOL_SIZE  to 1 and deflate algorithm to BEST_SPEED,    we have observed the each hour beginning's cpu spikes were relieved ,   and the user request latency is more stable than before

 

Any comments are welcome



--
This message was sent by Atlassian JIRA
(v7.3.1#73012)