[slf4j-user] Recommendation for library writers: use thiscopy and paste wrapper around SLF4j to avoid default static initialization

Sun Mar 19 01:26:24 CET 2017

Funny we do use Flume, and Kafka. The AMQP is just the start of the
pipe and that is because AMQP clients are everywhere (we do not use
JMS).

Besides.... RabbitMQ is plenty fast and in some cases faster than
Kafka. Right now I'm watching one of our Rabbit nodes do 40K/s. With
AMQP I can do all sorts of fancy stuff like drop a particular pattern
of messages or reroute them.

IMO Kafka protocol is not exactly super light either and the clients
aren't that good as well.

I'm not saying are architecture is the best but its certainly better
than many I have seen.

-Adam

On Sat, Mar 18, 2017 at 8:09 PM, Matt Sicker <boards at gmail.com> wrote:
> For pushing to ELK, why not use something more efficient like Kafka or
> Flume? ZeroMQ might be better there, too, but it's about as reliable as
> normal socket programming (i.e., in-memory messages are all lost in the
> event of a crash). I'm not a big fan of pulling in heavyweight messaging
> protocols or frameworks like JMS or AMQP nowadays for things like logging.
>
> On 18 March 2017 at 19:02, Adam Gent <adam.gent at snaphop.com> wrote:
>>
>> Matt,
>> We use a push model (pros and cons to both) with AMQP to ELK.
>>
>> But even if you use a pull model you may want to log out meta data (like
>> region or version info) which would come from a metadata service.
>>
>> We log like 30 or so attributes per event. I'll have to look how many come
>> from the metadata services.
>>
>> Sent from my iPhone
>>
>> On Mar 18, 2017, at 7:16 PM, Matt Sicker <boards at gmail.com> wrote:
>>
>> Kryo was already brought up, but on Android, if no-op logging is somehow
>> affecting performance (is Android's JVM *that* terrible that it can't
>> optimize that away? or is that an issue with the various ARM CPUs not doing
>> branch prediction well enough?), then just stick to using Minlog from Kryo.
>>
>> As for the error message on startup: it's definitely an error, not a
>> warning. An error indicates that a thing is broken, and in this case, that
>> means you forgot a dependency or you're in some strange ClassLoader
>> environment that didn't link the two libraries together properly. Log4j2 has
>> a similar error message.
>>
>> Now I know this doesn't solve everyone's use case, but for all you users
>> depending on Zookeeper or whatever to get logging configuration info, have
>> you considered just writing logs to stdout and collecting those log messages
>> via something like graylog? It's a lot easier to make things consistent
>> across dozens or hundreds of instances that way.
>>
>> On 18 March 2017 at 17:24, Adam Gent <adam.gent at snaphop.com> wrote:
>>>
>>> Like I said in another thread I concede particularly on the
>>> performance part. You have successful bashed me into thinking it was a
>>> terrible idea to bring up. Your points are all valid and I was more on
>>> your side than you think. I also frankly don't have the time to make
>>> terribly compelling arguments ... nor your apparent skill.
>>>
>>> However I do think I have a strong understanding of what SLF4J does (I
>>> almost take that particularly comment as an insult but I said you had
>>> stockholm syndrome sooo anywhoo :P).
>>>
>>> BTW the library isn't setting an application-wide policy its just
>>> ignoring the one the users is picking unless they explicit tell it not
>>> to (I used the serviceloader but you could use system properties or
>>> any  many other things... but I concede this is generally just making
>>> shit worse).
>>>
>>> > I'm still waiting to see one of those 20% cases...
>>>
>>> I guess I failed to explain the configuration requiring logging issue
>>> or you are just ignoring that part. It just feels wrong to make
>>> Appenders or whatever logging framework specific code to do that sort
>>> of initialization. And yes I understand the event replay.
>>>
>>> How do you configure your logging framework? We have code that presets
>>> System.properties (since sadly system.properties is the universal
>>> config) before the logging framework. Do you just hard code it and
>>> rely on some external mechanism for collecting logs? Remember I'm
>>> dealing with several logging backends.
>>>
>>>
>>> On Sat, Mar 18, 2017 at 5:57 PM, Joachim Durchholz <jo at durchholz.org>
>>> wrote:
>>> > Am 18.03.2017 um 20:04 schrieb Adam Gent:
>>> >>
>>> >> I generally agree with your points but I also know there are plenty of
>>> >> people that completely disagree.
>>> >> Just check out
>>> >>
>>> >> https://bill.burkecentral.com/2012/05/22/write-your-own-logging-abstraction/
>>> >
>>> >
>>> > That post motivates writing yet another SLF4J replacement to get rid of
>>> > both
>>> > SLF4J and whatever backends are being used, reinventing the SLF4J
>>> > wheel.
>>> > He's writing a logger facade to end all logger facades -
>>> > https://xkcd.com/927/ comes to mind.
>>> >
>>> > Or maybe he's writing his own logger library. In which case all the
>>> > 3rd-party libraries he's using are still happily logging to whatever
>>> > (now
>>> > left-unconfigured) backend they always have been logging to.
>>> >
>>> >> This would be an effort to some what mitigate those who have concerns
>>> >> have SLF4J and its depenency.
>>> >
>>> >
>>> > The post actually focuses on dependency hell.
>>> > Except it's no hell with SLF4J - newer versions are generally
>>> > API-compatible
>>> > with older versions, so you simply use the newest version of SLF4J.
>>> > And you use whatever backend you want, be it JUL, commons logging,
>>> > log4j, or
>>> > even logback - SLF4J can talk to all of them, in whatever version
>>> > you're
>>> > throwing at it.
>>> >
>>> >> I also wonder if your general opinion isn't stockholm syndrom.
>>> >
>>> >
>>> > Nope.
>>> > In fact I dislike a few things about SLF4J, but they are different from
>>> > your
>>> > pain points.
>>> > My impression is that you're working off some very basic
>>> > misunderstanding of
>>> > how SLF4J works, but I can't put my finger on it. I've been trying to
>>> > sort
>>> > that out, but whenever I explain something, you continue with a totally
>>> > unrelated topic and one message later you still stick with the idea I
>>> > tried
>>> > to explain away two messages ago. It's a bit frustrating, and I may
>>> > give up
>>> > on that - maybe somebody else has more luck.
>>> >
>>> >> Do you have any idea how many people hate:
>>> >>
>>> >>
>>> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>> >> further details.
>>> >
>>> >
>>> > Yeah I hated that, too.
>>> >
>>> >> As a library writer it is fairly annoying to have to deal with users
>>> >> complaining about.
>>> >
>>> >
>>> > WTF?
>>> > If I'm an application writer, I visit the URL given, read that all I
>>> > need to
>>> > do is to explicitly add a backend adapter to the class path, then I do
>>> > it.
>>> > If I'm a library writer and application writers come to me complaining,
>>> > I
>>> > tell them to Read The Friendly Error Message and buzz off.
>>> >
>>> > Really. It's SLF4J telling you you misconfigured it, and it's going to
>>> > do
>>> > its best but please fix that.
>>> > Maybe SLF4J should mark this as a WARNING so people don't get scared if
>>> > they
>>> > see "failed to load class", though the second line of text actually
>>> > tells
>>> > you what's going on.
>>> >
>>> > I hated seeing that message, but the fix was just a dependency in the
>>> > main
>>> > application's(!) dependencies (you don't add it to a library, nope, no,
>>> > never - it could be used as a scope=test dependency but that's about
>>> > it).
>>> >
>>> >>> How would the library know whether it's called in Android?
>>> >>> Should it even care?
>>> >>
>>> >>
>>> >> Yes. The reality is often it has to. Maybe not the code itself but the
>>> >> library maintainers might have to be concerned with Android (this
>>> >> includes backporting at times).
>>> >
>>> >
>>> > If the code does not have to care, then it shouldn't configure the
>>> > logging
>>> > backend.
>>> >
>>> >> Of course if you don't care about Android...
>>> >
>>> >
>>> > Oh I do.
>>> >
>>> >>> Your proposal does not change anything about the performance of trace
>>> >>> messages.
>>> >>
>>> >>
>>> >> Yes it does! I have benchmarked it myself... but alas I don't feel
>>> >> like pulling JMH out now nor the hassle of dealing with Android to
>>> >> prove such.
>>> >
>>> >
>>> > Benchmarking is notoriously tricky, and it is very, very easy to
>>> > measure
>>> > something totally different than what you think.
>>> > So to make sure that your claim is based on actual performance, you
>>> > need to
>>> > have somebody independent double-check your test code and your
>>> > measurements.
>>> >
>>> >> Unless you are extremely careful about configuration or
>>> >>
>>> >> just always globally using the NOPLogger there is generally some
>>> >> overhead the logging framework will do per call.
>>> >
>>> >
>>> > Since the noop looger does nothing, it's pretty impossible to have
>>> > extra
>>> > overhead.
>>> >
>>> >> Now I admit this is a
>>> >>
>>> >> while ago and I can't remember what logging framework was underneath.
>>> >> Maybe logback does a better job. I can't imagine it some how has
>>> >> specially loggers for each level (or maybe it does).
>>> >
>>> >
>>> > As I already explained elsewhere, it has just a small three-field
>>> > configuration object per "logger".
>>> > Initialization and configuration are done just once, anything else
>>> > would be
>>> > madness.
>>> >
>>> >>> That's a solved problem: SLF4J is collecting log messages during
>>> >>> startup,
>>> >>> and emitting them once the backend is ready.
>>> >>
>>> >>
>>> >> That doesn't fix anything. I have to now figure out how to intercept
>>> >> the backend before it does whatever you know like connecting to a
>>> >> message queue.
>>> >
>>> >
>>> > Why do you still need to do that?
>>> >
>>> >> Really... I don't think you realize how successful you guys have been
>>> >> in getting everyone to use SL4J.
>>> >
>>> >
>>> > That's unrelated - logging has been a thing long before even Java
>>> > entered
>>> > the scene.
>>> > It's just that with Java, writing servers became commonplace, and since
>>> > you
>>> > can't usefully run a debugger on a server, logging is your substitute.
>>> >
>>> >> Application startup time is a big deal... it is the number one hate of
>>> >> Java I hear most.
>>> >
>>> >
>>> > Yeah, but SLF4J is pretty innocent.
>>> > Swing is pretty slow to start, so interactive software used to take a
>>> > looong
>>> > time to start up.
>>> > Plus with Hotspot (i.e. the JVM that's standard everywhere except
>>> > Android),
>>> > any application starts fully interpreted, has the JIT compiler eat CPU
>>> > cycles in the background, and only after a while it becomes fast.
>>> >
>>> >> But you missed my point that most library consumers (applications)
>>> >> really just don't give crap about what their libraries are logging.
>>> >> And when they really really do they can go to the libraries web site
>>> >> and figure out how to turn on logging. I mean you have to figure out
>>> >> the logger names anyway.
>>> >
>>> >
>>> > Yeah, but my point is that the library shouldn't try to set an
>>> > application-wide policy. It shouldn't even set a logging policy for
>>> > itself,
>>> > because it cannot know what context it is running in.
>>> >
>>> >> My point was I only care about the parent framework and my
>>> >> applications codes logging. It seems a shame to have innocuous
>>> >> libraries  instantiating 1-3 different libraries indirectly (depending
>>> >> on how many wrappers, bridges, and converters you are using). It is
>>> >> just the kind of thing people complain about Java.
>>> >
>>> >
>>> > Actually SLF4J cuts down on that: you have only one backend, not three.
>>> > This also means you have only one configuration, so startup becomes
>>> > faster.
>>> >
>>> >> All in all I was *trying* to improve the adoption of SLF4J and have it
>>> >> handle more than 80/20 but 100 of most use cases.
>>> >
>>> >
>>> > I'm still waiting to see one of those 20% cases...
>>> > _______________________________________________
>>> > slf4j-user mailing list
>>> > slf4j-user at qos.ch
>>> > http://mailman.qos.ch/mailman/listinfo/slf4j-user
>>>
>>>
>>>
>>> --
>>> CTO
>>> SnapHop (snaphop.com)
>>> (twitter) @agentgt (linkedin) http://www.linkedin.com/in/agentgt
>>> (cell) 781-883-5182
>>> _______________________________________________
>>> slf4j-user mailing list
>>> slf4j-user at qos.ch
>>> http://mailman.qos.ch/mailman/listinfo/slf4j-user
>>
>>
>>
>>
>> --
>> Matt Sicker <boards at gmail.com>
>>
>> _______________________________________________
>> slf4j-user mailing list
>> slf4j-user at qos.ch
>> http://mailman.qos.ch/mailman/listinfo/slf4j-user
>>
>>
>> _______________________________________________
>> slf4j-user mailing list
>> slf4j-user at qos.ch
>> http://mailman.qos.ch/mailman/listinfo/slf4j-user
>
>
>
>
> --
> Matt Sicker <boards at gmail.com>
>
> _______________________________________________
> slf4j-user mailing list
> slf4j-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/slf4j-user

-- 
CTO
SnapHop (snaphop.com)
(twitter) @agentgt (linkedin) http://www.linkedin.com/in/agentgt
(cell) 781-883-5182