[logback-user] how to graceful shutdown logback with AsyncAppenders and without immediateFlush

[Michael Reinhold]

Hi Gena,

Again, I agree with most of your points at a high-level and don't doubt
that your solution can work.

I do disagree with your comment about the main requirement of logging
subsystems: it does depend on application requirements. For instance, in my
application, speed is of utmost importance - I am willing to lose debug and
warning messages in favor of keeping application performance at its
maximum. The only messages that require full reliability for my application
are warning and error events (this model is supported by AsyncAppender's
current design with its discardingThreshold).

All that being said, I'm not quite sure why you are resistant to a
framework for implementing shutdown hooks. Is there a particular reason you
object to the user being able to select which shutdown mechanism should be
employed by Logback? I understand that you feel your solution covers most
or all cases, but is it not possible that the end user of Logback may
disagree and prefer to apply their own shutdown procedure or policy?

As a side question, how long would you estimate it would take to modify
logback-core (and/or the other logback components) to implement your
proposed solution? Are you currently working on implementing this
switchToFailSafe functionality or are you proposing that someone else from
the logback community develop it?

Thanks!

Regards,

Mike Reinhold




On Thu, Mar 20, 2014 at 6:00 AM, Gena Makhomed <gmm at csdoc.com> wrote:

> On 20.03.2014 2:45, Michael Reinhold wrote:
>
>  I feel like we are speaking past each other a little bit in that I am
>> not proposing any of the specific shutdown hooks discussed so far at a
>> theoretical level (without being fully thought through or developed,
>> they are of course flawed). Rather, I am proposing that my solution be
>> implemented as a framework for Logback, contributors to Logback, and/or
>> end users of Logback to build shutdown hooks that fulfill specific
>> needs. Some of the theoretical hooks discussed could be implemented,
>> knowing that they are not general solutions, but rather niche solutions
>> that may be acceptable for some applications. Then, leveraging Logback's
>> community of contributors (including you and me), more general solutions
>> can be developed and included as part of Logback.
>>
>
> framework for logback shutdown hooks just not need at all,
> if only two simple solutions cover all possible use cases:
>
> 1) LoggerContext.stop for case app shutdown without JVM shutdown
> 2) LoggerContext.switchToFailSafe for case app shutdown == JVM shutdown
>
> register logback shutdown hook, which switch logback from "normal"
> to "failsafe" mode have sense even for web-applications, because
> tomcat can be stopped via signal SIGTERM, and such shutdown hook
> will flush all AsyncAppenders and FileAppenders memory bufffers.
>
> so, it is even not need any configuration option for enabling such
> logback shutdown hook, it should be enabled always for all contexts.
>
>
>  Based on this back and forth discussion, I don't think there is any
>> resistance to your proposed solution as it has the potential to be a
>> reasonably generic shutdown hook that could be applicable to many common
>> use cases.
>>
>
> can you provide any case in which my proposed solution not applicable?
>
>
>  Furthermore, it could be implemented as one of the pluggable
>> classes usable by the ShutdownHookAction.
>>
>
> java code is trivial: run(){ LoggerContext.switchToFailSafeMode(); }
>
>
>  If your implementation is as
>> generic and effective in practice as you have described, it could
>> actually be the default ShutdownHook implementation. If for some reason
>> a user found that your solution was non-optimal for their application,
>> they could implement (or work with the Logback community to implement)
>> an alternate ShutdownHook that does meet their needs.
>>
>
> right now exists only three available alternatives:
>
> 1. do nothing and lost all event in various memory buffers
> 2. use LoggerContext.stop and lost all events after stopping
> 3. use LoggerContext.switchToFailSafeMode for maximum reliability
>
> can you provide any other alternatives for these three?
>
> if even theoretically other possible use cases not exists here,
> may be not need implement framework for alternatives right now?
>
>
>  I hope this helps explain my position adequately - I don't see our two
>> solutions as competing alternatives, but rather as complimentary
>> components of the shutdown cleanup functionality of Logback.
>>
>
> I am talking not about only shutdown cleanup, I propose more generic
> solution, just add new, fail-safe mode. Switching logback to fail-safe
> mode useful in logback shutdown hook to prevent messages lost,
> and also switching logback into fail-safe mode will be useful
> for low-memory conditions.
>
> for example, we have one huge desktop application with many plugins.
> I wrote memory monitor, which continuously monitor java heap size,
> and continuously provide "live set" size of java objects in heap.
> also this monitor have "low memory" handler, which called in case
> if free heap memory (max heap memory - live set size) less then 50 MiB.
> ("live set" size == min used heap size for rolling window of last 5 min)
>
> switching logback into fail-safe mode will be very useful
> for this handler too, for preventing logback events lost in case
> if application will be stopped via forthcoming OutOfMemoryError.
>
> if AsyncAppenders not used and all FileAppenders configured with
> "immediateFlush = true" mode - for this special case "normal"
> and "fail-safe" mode will be identical. for other cases - not.
>
>
>  My portion
>> provides a framework for creating shutdown hooks that cleanup the
>> Logback context (possibly including some simple implementations) and
>> your portion includes a more complex, but fairly general implementation
>> of that shutdown hook framework. This gives us the best of both worlds -
>> the flexibility to plug different hooks in if needed as well as a solid
>> default implementation that should cover most use cases.
>>
>
> There exists only one way for
> "cleanup the Logback context" - just call LoggerContext.stop method.
>
> there is only one question - when do this cleanup. and only one strict
> answer, when LoggerContext.stop is required: this is case if application
> shutdown != JVM shutdown. In other cases "cleanup the Logback context"
> is not required at all, and switching logback into fail-safe mode
> is always better approach, because it prevent undesirable events lost.
>
> if user required to stop logging before actual JVM shutdown
> - they always can manually call LoggerContext.stop in own code.
>
> again, no need any special framework, LoggerContext.stop already exists.
>
> what you really need - it is configurable timeout
> for stopping AsyncAppender worker thread, if AsyncAppender
> used as frontend for appender working with remote logging service
> and because now in logback not exists any way for making reliable
> logging to remove/cloud logging services without log events lost.
>
> partial solution of remote logging is to play with timeout value
> and try to find acceptable tradeoff between application shutdown speed
> and application event lost rate from memory buffer on app shutdown.
>
> optimal solution - just create reliable async appender for remote
> services, which buffer events on disk and provides fast application
> shutdown and also provides reliable events delivery to remote logging
> service even in cases of partial network troubles/failures/overloads.
>
> main requirement for any logging subsystem: do not lost messages.
> all other wishes (speed,usability,etc) is secondary requirements.
>
>
>  This would seem
>> to fit with the methodology of design that Logback generally uses where
>> multiple implementations are possible and generally speaking a few
>> options are included in the library, but a default implementation that
>> covers most scenarios is also included in the library and, unless
>> specified otherwise, is the preferred option.
>>
>
> I can't imagine how multiple implementations of shutdown hook
> can better help prevent event lost if already exists almost
> perfect and safe algorithm for graceful shutdown of logback
> with minimal possible events lost via new "fail-safe" mode.
>
>
>  Once again, thank you for the good discussion!
>>
>
> You are welcome!
>
>
> --
> Best regards,
>  Gena
> _______________________________________________
> Logback-user mailing list
> Logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20140320/919b9982/attachment.html>