[logback-dev] RFC 5424 and Structured Data support.

[Ralph Goers]

On Dec 2, 2009, at 11:37 PM, Ceki Gulcu wrote:

> 
> Ralph Goers wrote:
> > On Dec 4, 2009, at 2:12 PM, Ceki Gülcü wrote:
> >
> >> Hi Ralph,
> >>
> >> I've looked at the changed you made in relation to RFC 5424. It's
> >> quite a significant architectural change. The RFC 5424 route is in the
> >> opposite direction of logback where one creates a new module for each
> >> new event type.
> 
> > I'm confused. What is this "module" that one creates? I assume you
> > mean an event type is a Marker? If not, what embodies it?
> 
> I was referring to modules like logback-classic for ILoggingEvent,
> logback-access for AccessEvent, AuditEvent for logback-audit.

Oh. I never understood the point of logback-access or logback-audit. It just seems like a nice way to make things very complicated. I've glanced at both but never figured out how they could be useful to me. I don't understand what benefit AuditEvent has over LoggingEvent if LoggingEvent can contain some sort of structured data object (as in EventData).  I understand why it is great to provide support for Jetty and Tomcat but it has always seemed odd to me that the support isn't done as a layer on top of logback-classic instead of logback-core. But then, I've never had a good reason to really look very hard at the code in logback-access.

> 
> >> This offers the benefit of type safety but has
> >> significant overhead from a packaging pov, whereas the 5424 route has
> >> little packaging overhead but omits type safety.
> 
> 
> >> Type safety of what? Everything is a String. With the latest changes
> >> Joern suggested that is enforced.
> 
> I have not seen Joern's changes. By using modules you can access each
> field directly with getters instead of get(eventFieldId). Many you
> write components this makes a palpabable difference.

Again, having to write a whole new module per event type just isn't practical. But then, I don't think you really mean "event type". logback-access and logback-audit certainly can handle more than one kind of "event". 

> 
> >> The RFC 5424 approach is not to be discarded. However, I don't think
> >> it's a route we should take at this *immediate* juncture. I propose to
> >> put it aside for the time being and come back to it, possibly with a
> >> vengeance, some time later.
> 
> > Obviously I have no control over what gets adopted into SLF4J/Logback
> > but I find this rather disappointing as it means I will either have to
> > hack something messy on top of Logback or finally get off my butt and
> > work on Log4j 2.0 which will require far more time than I want to
> > spend. Structured data support (RFC 5424) is an absolute requirement
> > for my project.
> 
> If structured data support is that important, them I encourage you
> continue to develop your fork. SLF4J is changing very slowly so
> keeping in sync should not be too difficult.  Logback is a different
> matter, but if your modifications are localized, then rebasing
> with the logback trunk should be possible as well.
> 

The main issue I faced was there is no good way to pass a data object through SLF4J to the underlying logging implementation. Really, the most significant change I made was to add XLocationAwareLogger to SLF4J to accomplish that, and even that is fairly transparent. I also still haven't looked at Joern's code, but if he is generalizing that to a more generic Message interface then that could be a very good thing. Within Logback the main issue was determining if the first parameter is a StructuredData object. At first I tried adding it to the LoggingEvent but soon realized that would be far more extensive a change. Again, I'm interested to see what Joern has done.

One of the main reasons I am enamored with RFC 5424 is that it is a fabulous way of sending serialized event data around without being tied to any particular logging implementation, with the exception of the names of the keys. I don't think sending a throwable with its call stack would be a good idea, but virtually everything else could be captured in structured data such as the mdc (which I am already doing) and even the caller data.  I don't really think of it as a syslog message but as an industry standard way of sending event messages through a system. Using a message bus one can easily inspect the data as it passes through the bus. Plus, I just noticed yesterday that there are two new RFCs, 5674 - Alarms in Syslog and 5675 - Mapping SNMP Notifications to Syslog Messages, that are building on top of the 5424 spec and making it even more useful.

As far as developing on my fork, there is no way I can have my development teams use SLF4J and Logback from my private branches. Unfortunately, at this point all I can do is build a kludge around EventData - which is more expensive because it has to be serialized and deserialized in the Appender - and format that as structured data in my own IETFSyslogLayout class. Plus, the fact that the JMSAppender won't use a layout also means I have to write my own version of that or the IETFSyslogAppender if I use that directly instead of JMS. All of this code would then have to reside in my own source control system at Intuit which is exactly what I am hoping to avoid.


Ralph