[logback-user] getting logback to scale on multitenant app

[Jamie]

David

Thanks for your consideration. You seem to imply this shouldn't be 
happening. If so, that's great. The below initAppenders() function is 
called on startup of the server. Perhaps the way the appenders are being 
initialized and startedis causing the behavior. Is there an alternative?

private void initAppenders() {

       RollOverAppenderSettings debugAppender = 
AppendersFactory.getAppenderSettings(APPENDER.DEBUG);
       assert debugAppender != null;
       debugAppender.setLoggerName(application.getLoggerName());
       debugAppender.setLogFile(getDebugLogFile());
       LogUtil.addAppender(debugAppender);

       RollOverAppenderSettings auditAppender = 
AppendersFactory.getAppenderSettings(APPENDER.AUDIT);
       assert auditAppender != null;
       auditAppender.setLoggerName("audit-" + application.getLoggerName());
       auditAppender.setLogFile(getAuditLogFile());
       LogUtil.addAppender(auditAppender);
       LogUtil.applyLogLevel("audit-" + application.getLoggerName(), 
"info");

       RollOverAppenderSettings trashAppender = 
AppendersFactory.getAppenderSettings(APPENDER.TRASH);
       assert trashAppender != null;
       trashAppender.setLoggerName("trash-" + application.getLoggerName());
       trashAppender.setLogFile(getTrashLogFile());
       LogUtil.addAppender(trashAppender);
       LogUtil.applyLogLevel("trash-" + application.getLoggerName(), 
"info");
}

Where the following methods are defined in LogUtil:

public static void addAppender(RollOverAppenderSettings appender) {
    LoggerContext loggerContext = (LoggerContext) 
LoggerFactory.getILoggerFactory();

    File logFile = appender.getLogFile();

     RollingFileAppender<ILoggingEvent> rfAppender = new 
RollingFileAppender<>();
     rfAppender.setName(appender.getLoggerName());
     rfAppender.setContext(loggerContext);
     rfAppender.setFile(logFile.getPath());

     RollingPolicy rollingPolicy = appender.getRollingPolicy();
     rollingPolicy.setParent(rfAppender);
     rollingPolicy.start();

     TriggeringPolicy<ILoggingEvent> triggeringPolicy = 
appender.getTriggeringPolicy();
     triggeringPolicy.start();

     PatternLayoutEncoder encoder = new PatternLayoutEncoder();
     encoder.setContext(loggerContext);
     encoder.setPattern(appender.getLogEntryPattern());
     encoder.start();

     rfAppender.setEncoder(encoder);
     rfAppender.setRollingPolicy(rollingPolicy);
     rfAppender.setTriggeringPolicy(triggeringPolicy);
     rfAppender.start();

     AsyncAppender asyncAppender = new AsyncAppender();
    asyncAppender.setName(appender.getLoggerName());
    asyncAppender.setContext(loggerContext);
     asyncAppender.setQueueSize(1024);
     asyncAppender.addAppender(rfAppender);
     asyncAppender.start();

        ch.qos.logback.classic.Logger logbackLogger = 
loggerContext.getLogger(appender.getLoggerName());
        logbackLogger.setAdditive(false);
     logbackLogger.addAppender(asyncAppender);

     if (System.getProperty("stdout") != null) {
         ConsoleAppender<ILoggingEvent> consoleAppender = new 
ConsoleAppender<>();
         PatternLayoutEncoder stdoutEncoder = new PatternLayoutEncoder();
         stdoutEncoder.setContext(loggerContext);
         stdoutEncoder.setPattern("%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] 
%m%n");
         stdoutEncoder.start();
         consoleAppender.setEncoder(stdoutEncoder);
         consoleAppender.start();
         logbackLogger.addAppender(consoleAppender);
     }

     StatusPrinter.print(loggerContext);
    logger = LoggerFactory.getLogger(appender.getLoggerName());
}

public static void applyLogLevel(String loggerName, String logLevel) {
    Logger log = LoggerFactory.getLogger(loggerName);
    if (!(log instanceof ch.qos.logback.classic.Logger)) {
       logger.debug("cannot set log level. not using logback");
       return;
    }
    ch.qos.logback.classic.Logger logger = 
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger(loggerName);
    Level level = Level.INFO;
    try {
       level = Level.valueOf(logLevel.toUpperCase(Locale.ENGLISH));
    } catch (Exception e) {
       logger.error("invalid new log level value {level='" + logLevel+ 
"'}");
    }
    logger.setLevel(level);
}
On 2017/12/15 00:04, David Roussel wrote:
> Hi Jamie,
>
> It’s not clear how your “multi-tennent” approach affects the loading of logback.  I guess somehow you are creating new instances of logback?
>
> Each instance of AsyncAppender is blocking, and thus requires one thread per per AsyncAppender.
>
> Dave
>
>> On 14 Dec 2017, at 21:28, Jamie <jamie at stimulussoft.com> wrote:
>>
>> Hi
>>
>> I'd appreciate guidance on  how to get Logback to use less resources in our multitenant app. It seems every time we add a new tenant, log back creates a few more async appender threads. This affects the scalability of our software. In our app, each tenant has an audit and a debug log. Async apppender is being used to write to each log respectively. Our problem is that whenever a new tenant is added, many more threads get created. it seems a new  worker thread is assigned to each appender.  Is there a way to get Logback to rather use a shared thread pool?
>>
>> Stacks at 2017-12-14 10:33:57 PM (uptime 22m 24s)
>>
>> AsyncAppender-Worker-audit-template [WAITING] [DAEMON]
>> sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
>> java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() AbstractQueuedSynchronizer.java:2039
>> java.util.concurrent.ArrayBlockingQueue.take() ArrayBlockingQueue.java:403
>> ch.qos.logback.core.AsyncAppenderBase$Worker.run() AsyncAppenderBase.java:289
>>
>> Thanks in advance
>>
>> Jamie
>>
>>
>>
>> _______________________________________________
>> logback-user mailing list
>> logback-user at qos.ch
>> http://mailman.qos.ch/mailman/listinfo/logback-user
> _______________________________________________
> logback-user mailing list
> logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user

-- 

Stimulus Software


Stimulus Software
Office: +1-713-343-8824 / Fax: +1 713 357 6647
http://www.stimulussoft.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20171215/6100877f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smalllogo.png
Type: image/png
Size: 5495 bytes
Desc: not available
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20171215/6100877f/attachment-0001.png>