[logback-user] how to graceful shutdown logback with AsyncAppenders and without immediateFlush

[Gena Makhomed]

On 20.03.2014 2:45, Michael Reinhold wrote:

> I feel like we are speaking past each other a little bit in that I am
> not proposing any of the specific shutdown hooks discussed so far at a
> theoretical level (without being fully thought through or developed,
> they are of course flawed). Rather, I am proposing that my solution be
> implemented as a framework for Logback, contributors to Logback, and/or
> end users of Logback to build shutdown hooks that fulfill specific
> needs. Some of the theoretical hooks discussed could be implemented,
> knowing that they are not general solutions, but rather niche solutions
> that may be acceptable for some applications. Then, leveraging Logback's
> community of contributors (including you and me), more general solutions
> can be developed and included as part of Logback.

framework for logback shutdown hooks just not need at all,
if only two simple solutions cover all possible use cases:

1) LoggerContext.stop for case app shutdown without JVM shutdown
2) LoggerContext.switchToFailSafe for case app shutdown == JVM shutdown

register logback shutdown hook, which switch logback from "normal"
to "failsafe" mode have sense even for web-applications, because
tomcat can be stopped via signal SIGTERM, and such shutdown hook
will flush all AsyncAppenders and FileAppenders memory bufffers.

so, it is even not need any configuration option for enabling such
logback shutdown hook, it should be enabled always for all contexts.

> Based on this back and forth discussion, I don't think there is any
> resistance to your proposed solution as it has the potential to be a
> reasonably generic shutdown hook that could be applicable to many common
> use cases.

can you provide any case in which my proposed solution not applicable?

> Furthermore, it could be implemented as one of the pluggable
> classes usable by the ShutdownHookAction.

java code is trivial: run(){ LoggerContext.switchToFailSafeMode(); }

> If your implementation is as
> generic and effective in practice as you have described, it could
> actually be the default ShutdownHook implementation. If for some reason
> a user found that your solution was non-optimal for their application,
> they could implement (or work with the Logback community to implement)
> an alternate ShutdownHook that does meet their needs.

right now exists only three available alternatives:

1. do nothing and lost all event in various memory buffers
2. use LoggerContext.stop and lost all events after stopping
3. use LoggerContext.switchToFailSafeMode for maximum reliability

can you provide any other alternatives for these three?

if even theoretically other possible use cases not exists here,
may be not need implement framework for alternatives right now?

> I hope this helps explain my position adequately - I don't see our two
> solutions as competing alternatives, but rather as complimentary
> components of the shutdown cleanup functionality of Logback.

I am talking not about only shutdown cleanup, I propose more generic
solution, just add new, fail-safe mode. Switching logback to fail-safe
mode useful in logback shutdown hook to prevent messages lost,
and also switching logback into fail-safe mode will be useful
for low-memory conditions.

for example, we have one huge desktop application with many plugins.
I wrote memory monitor, which continuously monitor java heap size,
and continuously provide "live set" size of java objects in heap.
also this monitor have "low memory" handler, which called in case
if free heap memory (max heap memory - live set size) less then 50 MiB.
("live set" size == min used heap size for rolling window of last 5 min)

switching logback into fail-safe mode will be very useful
for this handler too, for preventing logback events lost in case
if application will be stopped via forthcoming OutOfMemoryError.

if AsyncAppenders not used and all FileAppenders configured with
"immediateFlush = true" mode - for this special case "normal"
and "fail-safe" mode will be identical. for other cases - not.

> My portion
> provides a framework for creating shutdown hooks that cleanup the
> Logback context (possibly including some simple implementations) and
> your portion includes a more complex, but fairly general implementation
> of that shutdown hook framework. This gives us the best of both worlds -
> the flexibility to plug different hooks in if needed as well as a solid
> default implementation that should cover most use cases.

There exists only one way for
"cleanup the Logback context" - just call LoggerContext.stop method.

there is only one question - when do this cleanup. and only one strict
answer, when LoggerContext.stop is required: this is case if application
shutdown != JVM shutdown. In other cases "cleanup the Logback context"
is not required at all, and switching logback into fail-safe mode
is always better approach, because it prevent undesirable events lost.

if user required to stop logging before actual JVM shutdown
- they always can manually call LoggerContext.stop in own code.

again, no need any special framework, LoggerContext.stop already exists.

what you really need - it is configurable timeout
for stopping AsyncAppender worker thread, if AsyncAppender
used as frontend for appender working with remote logging service
and because now in logback not exists any way for making reliable
logging to remove/cloud logging services without log events lost.

partial solution of remote logging is to play with timeout value
and try to find acceptable tradeoff between application shutdown speed
and application event lost rate from memory buffer on app shutdown.

optimal solution - just create reliable async appender for remote
services, which buffer events on disk and provides fast application
shutdown and also provides reliable events delivery to remote logging
service even in cases of partial network troubles/failures/overloads.

main requirement for any logging subsystem: do not lost messages.
all other wishes (speed,usability,etc) is secondary requirements.

> This would seem
> to fit with the methodology of design that Logback generally uses where
> multiple implementations are possible and generally speaking a few
> options are included in the library, but a default implementation that
> covers most scenarios is also included in the library and, unless
> specified otherwise, is the preferred option.

I can't imagine how multiple implementations of shutdown hook
can better help prevent event lost if already exists almost
perfect and safe algorithm for graceful shutdown of logback
with minimal possible events lost via new "fail-safe" mode.

> Once again, thank you for the good discussion!

You are welcome!

-- 
Best regards,
  Gena