[logback-dev] Lumberjack infos - was: Re: Logback Chainsaw Bridge

[Joern Huxhorn]

Hi again.

First of all: I hope it's ok to post semi-offtopic stuff to the 
logback-developer-list. I'd like to keep this of the user list until I 
release a version incl. source.
If using the logback-developers-list for a message like this is 
inappropriate/a problem, please let me know.

You can find a new version (0.9.15) at 
http://lumberjack.huxhorn.de/lumberjack.zip
I fixed some caching stuff...

Maarten Bosteels wrote:
> Yesterday I tried out Lumberjack (cool name, by the way) which also
>   
Yes, I like that name, too, but http://lumberjack.sourceforge.net/ is 
already used by another logging-related project so I will most likely 
change it before official release.
> looks promising (it's not yet released). I have some questions for
> Joern Huxhorn, the author of Lumberjack
> * I don't understand why it is file-based instead of memory-based ?
> You probably have another usage pattern. For me it's enough if it
> would keep the last 20.000 events (like chainsaw does)  and keep those
> events in memory
>   
I'm using disk- instead of memory-based logging mainly because Chainsaw 
crashed with OutOfMemoryExceptions in case of heavy load. This resulted 
in freezes of our webapp, i.e. the webapp froze as long as chainsaw was 
still running! I fixed the freeze-problem in chainsaw (close socket in 
case of *any* throwable).and decided to write my own viewer in my 
spare-time...

One of my main objectives was the processing of a potentially very large 
numbers of events from multiple sources at the same time.
> * the search feature works great, but I would like an option to do
> live filters the same way
> (ie, without opening a new tab)
>
>   
If you press shift-enter in an already filtered view then the current 
filter is replaced with the new combined filter. This is probably 
(nearly) what you want. The problem with the disk-based approach is the 
same as it's main feature - large number of events. It's quite easy 
(fast) to filter 10.000 in-memory-events... but it's absolutely 
necessary to perform the filtering in a different view that is updated  
asynchronously in case the disk-based solution.
That's why at least one tab is required...

I plan to implement a filter-manager similar to the "Message Filters" 
dialog of Thunderbird but the release date is "when it's done" ;) I 
don't know if this will be implemented pre-1.0. I'll probably also 
implement groovy-filters so you can write your own complex filters in 
groovy.
> Maybe I will have a go at writing my own LogViewer GUI but I have very
> little Swing experience (nor SWT), and a constant lack of free time
> :-)
>
> Currently the SocketAppender relies on java serialization, but I'm
> wondering if it wouldn't
> be better to define a language independent protocol for
> encoding/decoding logging events.
> Then SocketAppenders for log4j, LogBack, log4cxx,.. could use the same
> protocol and everyone would benefit.
> That's how the XmlSocketAppender in log4cxx is able to work with
> Chainsaw, but I would
> prefer a binary protocol to mimimize bandwidth. (eg. Hessian)
>   
I'm gzipping the serialized log events in my log-files which saves about 
55%. This could also be done in SocketAppender... I wanted to implement 
it and submit a patch but I don't have enough spare-time left. I second 
that xml-events wouldn't hurt but an XmlSocketAppender should'nt be 
contained in the classic/access modules but in new modules instead to 
keep dependencies low (e.g. stax).
> Even it's only used by LogBack it would have benefits, changes to the
> serialized form of LoggingEvent wouldn't break older clients, and it
> might be faster than java serialization.
>
> One last thing: currently the SocketAppender is synchronous, slowing
> down the application
> when the network is slow, wouldn't it make sense to do the I/O in a
> separate thread ?
>   
The problem with asynchronous SocketAppenders is that you have 
essentially three options:
a) you keep events in an in memory-back-buffer. This will lead to 
out-of-memory situations if more events are produced than transfered. At 
this point your app will either explode or drop events. Both is not 
really an option.
b) you keep the events in a disk-based buffer.This will lead to 
out-of-disk-space situations if more events are produced than 
transfered. See a) ;)

So event transmission must be synchronous.

We sometimes have 5+ people receiving log-events from our test-server at 
the same time and the current SocketAppender pauses some seconds for 
every unavailable ip (Socket is opened without timeout) so I thought 
about a SocketAppender serving multiple ip's asynchronously while still 
being synchronous at event-level.

This would have the added benefit that the logging-event can be 
serialized (gzipped?) just once before sending the same bytes to all 
available sockets instead of doing it for every SocketAppender.
> Or create a MINA-based appender :-)
> Would that even work, since MINA itself depends on SLF4J ?
>
> Regards, and thanks for reading this far :-)
> Maarten
>
> [1] http://vigilog.sourceforge.net/index.html
>   
Nice! I didn't know about this project! I have to check it out...

Regards, and *also* thanks for reading this far :-)

Joern.