[slf4j-dev] Benefits of Message objects?

[John Vasileff]

On Sep 9, 2011, at 6:23 AM, Joern Huxhorn wrote:

> 
> On 08.09.2011, at 21:11, John Vasileff wrote:
> 
>> I put together the following examples to help net out benefits of supporting "Map supplementalData" arguments in Logger methods in addition to Message object support.
>> 
>> Support for supplementalData does not replace Message objects, but is perhaps more of a shortcut for common usage scenarios.  In the end, a Message object is always created by either the application writer or SLF4J.
>> 
>> 
>> Scenario #1: Working with a domain object "posting" that implements java.util.Map.  A 1.6.2 error log may look like:
>> 
>> logger.error("error parsing markdown '{}'", badString, e);
>> 
>> but we want to add all details of the "posting" domain object without including them in the message.  Using the supplementalData log methods:
>> 
>> // just add the data
>> Logger logger = LoggerFactory.getLogger(MyClass.class);
>> logger.error(posting, "error parsing markdown '{}'", badString, e);
>> 
>> // add the data, but also allow named parameter formatting
>> Logger logger = LoggerFactory.getLogger(MyClass.class).formatWith(NamedParamFormatter.getInstance());
>> logger.error(posting, "error parsing markdown for posting {id} '{}'", badString, e);
>> 
>> // just add the data, and format using java.util.formatter
>> Logger logger = LoggerFactory.getLogger(MyClass.class).formatWith(JavaUtilFormatter.getInstance());
>> logger.error(posting, "error parsing markdown '%s'", badString, e);
> 
> One huge problem with this approach is that Logger, for a given name/class, has to be a singleton. You absolutely must not change its state, otherwise chaos will happen.
> 
> I'll give you just one example:
> Suppose ThreadA is changing the Logger with formatWith. Then, before ThreadA executes its log statement, ThreadB is changing the same instance with a different formatWith call. This will cause ThreadA to log with a wrong formatting. A mutable Logger would automatically make it non-threadsafe.
> 
> I firmly believe that default formatting must not be configurable. Doing so would just introduce a big pile of problems.
> 

I agree 100% that formatting should never be changed once the logger is setup.  And I also see that the same instance of Logger is returned for a given name in, for example, Log4jLoggerFactory.  But if in the end the idea of formatWith() makes for a nice application facing API, I would hate to rule it out prematurely based on what may be implementation details.

Having said that, I don't really have a strong need for formatWith(...) - java.util.Formatter seems unnecessary to me, so it would just be an added convenience for NamedParameter style formatting.  But I do feel that whether in 2.0, 3.0, or whenever, the ability to add this type of feature without breaking binary compatibility is very valuable.  So I think it is worth netting out, even if the consensus is that the particular feature of overriding the formatter isn't necessary or desired for 2.0.

>> 
>> If we used Message objects:
>> 
>> Logger logger = LoggerFactory.getLogger(MyClass.class);
>> 
>> // just add the data
>> logger.error(new DefaultMessage(posting, "error parsing markdown '{}'", badString, e));
>> 
>> // add the data, but also allow named parameter formatting
>> logger.error(new NamedParamFormattingMessage(posting, "error parsing markdown for posting {id} {}", badString, e));
>> 
>> // just add the data, and format using java.util.formatter
>> logger.error(new JavaUtilFormatterMessage(posting, "error parsing markdown for posting %s", badString, e));
>> 
>> 
>> Scenario #2: Support for StructuredData.  The StructuredData object may augment a log message, or both augment the message and provide the message with toString().
>> 
>> Using SupplementalData methods, where SD implements Map
>> 
>> StructuredData sd = new StructuredData(...);
>> logger.error(sd);
>> logger.error(sd, "some log message"); 
>> 
>> Using Message object methods, where SD implements Message
>> 
>> StructuredData sd = new StructuredDataMessage(...);
>> logger.error(sd);
>> StructuredData sd = new StructuredDataMessage(..., "some log message");
>> logger.error(sd);
>> 
> 
> The idea was to put both the internal structure and the formatting in the respective Message implementation, with ParameterizedMessage being the implementation for the default SLF4J formatting. A shortcut exists for this type of messages, namely the foo(String pattern, Object... args) methods, since this is the message type encouraged by the logging framework.
> 
> It is encouraged since it is a perfect mix of features (placeholder replacement) and speed (it is *much* faster than JUL and likely faster than most other implementation). It is also the type of formatting already documented by the API and accepted by its users.
> 
> The 99% of the users will just keep on using the "normal" logging methods and ignore the newly defined ones. This is the main target we are developing for.
> 
> The foo(Message message) type of methods, however, enable the 1% "pro users" to implement specific features they need - and the logging framework will be able to (optionally, you could filter on the Message class) cope with those arbitrary Message types in a default way or implement specialized handling if that is required.
> 
> Take the StructuredDataMessage ( http://bugzilla.slf4j.org/show_bug.cgi?id=148 ) as an example. It does not just define a Message with key-value pairs in a Map<String, String>. Instead, it is more specific. It is used to create messages that conform to ( http://tools.ietf.org/html/rfc5424 ). This involves making sure that the keys must have a maximum length of 32 characters. Such a requirement can't be met by the abstract logging interface. It is an implementation detail of RFC5424 Syslog messages.
> In addition to the Map<String, String>, such a StructuredDataMessage could implement the whole range of the specification, e.g. it could bypass the normal logging levels and make use of the Syslog severities instead.
> 

In the 99% group, there are some (me!) that would really like to keep everything as is, but also be able to quickly and conveniently add some key/value pair data to the log entry.

I was trying to show that this also supports bug 148.  StructuredData could implement Map<String,Object>.  Loggers that understand only Map<String,Object> would at least handle the parts that are exposed by the Map interface.  Loggers that understand StructuredData could provide 100% custom handling, such as creating a Log4j2 StructuredDataMessage from the StructuredData object.  In either case, key length validation, etc. would be handled by the StructuredData class.

I think it would be valuable to introduce supplementalData/namedParameters to the masses, while also providing things like StructuredData to advanced users.

To be clear: I am not suggesting we avoid Message/Entry in favor of supplementalData/namedParameters.  I think the Message concept is great and think we should have both.  Use supplementalData/namedParameters as a quick way to add data to an entry, or use Message/Entry to "take over" all data and formatting of the log entry.

> One should not forget that someone implementing his own message type could also define a wrapper around the SLF4J Logger to support efficient creation of his handcrafted Messages. A special SyslogLogger implementation could automatically match the Syslog severity to a proper SLF4J level, for example, while still using the whole logging backend without any wheel-reinvention.
> 
> Another example for specific needs would be end-user-facing localized messages. Why should I reinvent my own logging layer if I could just use SLF4J/<insert backend here> for that?
> 
> Audit logging is another use case.
> See http://audit.qos.ch/manual/index.html and http://audit.qos.ch/apidocs/ch/qos/logback/audit/AuditEvent.html 
> This could be implemented using a AuditMessage instead while still being able to use the full range of Logback-Classic appenders but with the ability to add specific appenders handcrafted for specific tasks like writing them into a special audit database.
> 
> One small, additional thing is the possibility of immutable Message instances that could be kept and reused if there are no variables involved.
> 

I think all of these would be supported - I am trying to build on your proposal.

For localized messages specifically, there may be a really nice way to handle this with "formatWith(...)" - I'll have to give it some thought.  But in any case, Message/Entry objects could always be used.  In any case, I agree 100% that SLF4J should handle localization, otherwise we would be breaking pluggability of back end loggers.

> Concerning "COMPATIBILITY #1" in your previous message:
> It is absolutely crucial that the original interface is left unchanged so that both use and implementations of that interface will keep on working as expected. It also needs to stay 1.4 compatible and logging frameworks that do not require 1.5 should still just implement the old SLF4J interface. The new interfaces will be available automatically via wrapper in that case.
> 
> If the logging implementation requires Java >= 1.5 anyway (e.g. Logback) then it would be a good idea to implement the new interface instead, providing backwards-compatibility by means of a wrapper in the other direction.

I think we are in agreement here, and we definitely cannot break < 2.0 Logger implementations.  In 2.0, I think we can enhance org.slf4j.Logger, maintain compatibility, and even provide many of the new features to 2.0 api users that choose to use a < 2.0 logger.  If I am right, I think this is a huge advantage, both in improving the feature set for older Logger users, and in sticking with only one interface rather than making users move to a different one (and for new users, avoid confusion about two Loggers, etc.)

I'll provide more detail on this soon.

> 
> Concerning your Entry interface:
> The main idea of the very minimal Message interface is the absolute freedom it gives to the developer. A message does not have to have a pattern and parameters. Why should it? And why should I create an abstract base-class that returns null for all methods that are not applicable to that specific message/entry type? Isn't that enough evidence that the interface would be too specific?
> I did consider to add Throwable to the Message interface but ultimately decided against it for exactly that reason. It just doesn't make sense for every type of message imaginable.

My idea is to encapsulate all data and formatting for a log entry, and simplify all interfaces.  The Entry interface basically defines the fundamental data components we are providing to Logger implementations.  There aren't that many methods, and having them all here provides the ability for Entry classes to handle all data aspects of an entry, and clarity that this can be done.  I'll add more thoughts on this later.

> I'm also not a big fan of including either MDC or NDC in the Message since those belong to the LoggingEvent, similarly to the call-stack or the timestamp of the event. Neither of them are visible at the SLF4J interface level.
> 

I haven't spent much time looking at MDC/NDC, but if (somewhere) there is support for named parameters, it might be nice to do:

("user {mdcUserId} had a problem with {}", problemString)

A problem I have had with NDC before is that I have disabled logging NDC info to cut down on noise.  This would provide a mechanism to force a particular value into a message when it is relevant.  Just a thought.

> To finish this wall of text:
> I can not stress enough that the Message interface would enable us to *filter* on the specific type of Message in the logging configuration and/or appender implementation. Granted, a better place for this argument would be the Logback mailing list but it IS a major feature and does indeed open a huge door of possibilities for future enhancements, all without touching the logging interfaces again.
> 
> Joern
> _______________________________________________
> slf4j-dev mailing list
> slf4j-dev at qos.ch
> http://qos.ch/mailman/listinfo/slf4j-dev

Additional thoughts on StructuredData:  mapping to Syslog severities raises an interesting question.  Perhaps Event should also encapsulate Level and Marker.  I know you are suggesting a more simple Message object, and this pulls in the opposite direction, but it would really provide a clean design whereby an application could specify _only_ the syslog severity, and the StructuredData class could handle mapping to trace/debug/etc. levels.  The interface would simply be:

public log(Entry entry);

Entry would provide full power and encapsulation over individual log events.  If I had time I would try to rework some of my above comments with this in mind, but we are just sharing ideas, right?


John