[logback-user] AsyncAppenderBase not flushing queue during JVM shutdown

[David Roussel]

Did you try shutting down logback cleanly. Like this http://stackoverflow.com/questions/3678755/do-i-need-to-flush-events-when-shutting-down-using-logback

David

> On 5 Mar 2014, at 20:44, Michael Reinhold <mike at coursescheduler.io> wrote:
> 
> Hi Ceki, 
> 
> I am currently using the AsyncAppender in combination with the LogglyAppender from the Logback extensions project. While working on a number of aspects of my application, I found that I was not consistently getting all of the log messages that I expected. In particular, when the application shuts down immediately (or very shortly) after a burst of logging activity, the tail of those log events is often not present in Loggly. From a number of tests, this issue is not restricted to use with the LogglyAppender, but is simply more evident because of the latency involved.
> 
> Looking through the source code for the AsyncAppenderBase, I saw the following:
> 
> You create the Async sender thread as a Daemon thread
> 
> 
> addInfo("Setting discardingThreshold to " + discardingThreshold);
>     worker.setDaemon(true);
>     worker.setName("AsyncAppender-Worker-" + worker.getName());
>     // make sure this instance is marked as "started" before staring the worker Thread
>     super.start();
>     worker.start();
> 
> 
> In the sender thread, if you determine that the parent thread has stopped or the async sender thread has been interrupted, you allow the worker thread to flush remaining log events in the queue.
> 
> 
> while (parent.isStarted()) {
>         try {
>           E e = parent.blockingQueue.take();
>           aai.appendLoopOnAppenders(e);
>         } catch (InterruptedException ie) {
>           break;
>         }
>       }
> 
>       addInfo("Worker thread will flush remaining events before exiting. ");
>       for (E e : parent.blockingQueue) {
>         aai.appendLoopOnAppenders(e);
>       }
> 
>       aai.detachAndStopAllAppenders();
> 
> 
> From what I can tell, during JVM shutdown (for a standalone SE instance, the same is probably not true for EE instances with application servers) daemon threads may be terminated without allowing the the AsyncAppenderBase to escape the while loop and proceed onto the queue flush for loop. 
> 
> From Brian Goetz in Java Concurrency in Practice:
> "When a thread exits, the JVM performs an inventory of running threads, and if the only threads that are left are daemon threads, it initiates an orderly shutdown. When the JVM halts, any remaining daemon threads are abandoned finally blocks are not executed, stacks are not unwound the JVM just exits. Daemon threads should be used sparingly few processing activities can be safely abandoned at any time with no cleanup. In particular, it is dangerous to use daemon threads for tasks that might perform any sort of I/O. Daemon threads are best saved for "housekeeping" tasks, such as a background thread that periodically removes expired entries from an in-memory cache."
> 
> To test this, I inserted a break point in the AsyncAppenderBase at the call to addInfo and ran a variety of different scenarios. At no point in time was I able to get the breakpoint to stop at the addInfo line. 
> 
> I don't think there are any clear cut solutions to this. Making the worker thread a user thread instead of daemon thread has its own implications, particularly that if all other user threads have exited the async threads would keep the JVM instance alive (unless System.exit has been called, in which case I believe that you will still have lost log events even if the async processing thread is not a daemon). It might be possible to create a shutdown hook that does the queue flushing for the async worker - though you may need to consider the possibility of other shutdown hooks wanting to log events as well.
> 
> I'm currently mocking up a version of the AsyncAppenderBase and AsyncAppender that installs a shutdown hook as described previously. I think a "queue idle time" configuration might be the best way to handle other shutdown hooks adding log events (aka - after processing whatever was in the queue, if no new events are added within x ms then the shutdown hook can assume nothing else will be adding log events and can exit).  An alternative might be to have the shutdown hook query the ThreadMXBean API to determine if other shutdown hooks are still executing and possibly adding log events (though the threads that are expected to be running during shutdown may be not only application specific but also JVM implementation specific... I'm not sure).
> 
> Let me know what you think. I'll let you know if I feel that my mockup may be viable...
> 
> Regards,
> 
> Mike Reinhold
> 
> _______________________________________________
> Logback-user mailing list
> Logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20140305/83261d74/attachment-0001.html>