[logback-dev] [qos-ch/logback-decoder] f9b87f: ongoing work on layout-pattern-to-regex converters...

[ceki]

On 19.06.2012 23:42, Tony Trinh wrote:
> On Tue, Jun 19, 2012 at 2:37 PM, ceki <ceki at qos.ch <mailto:ceki at qos.ch>>
> wrote:
>
>
>     Hi Tony,
 >
> I piggy-backed the PatternLayoutBase class to reuse its converter logic
> for converting the layout patterns into regular expressions. I don't
> think this is a very clean way of doing it, but I went with it for now.


Using PatternLayoutBase logic is OK although piggy-backing on the just
parser logic is probably a little better. I would suggest to use the
code in the start() method of PatternLayoutBase. See lines 80 to 91 of
PatternLayoutBase. In essence, the code boils down to:

Converter<E> head = null;
Parser<E> p = new Parser<E>(pattern);
Node t = p.parse();
head = p.compile(t, someConverterMap);
ConverterUtil.startConverters(head);


There are differences between the code shown above and what the
start() method of PatternLayoutBase does. The start() method sets the
context of the converters. However, for logback-decoder the context
has to be different than LoggingContext. A ContextBase instance could
do. The other difference is that start() invokes post compile
processing. In logback-classic, this is just adding %xEx converter at
the end of the pattern if no converter deals with throwables. To
simplify the work of the decoder, I think logback-proper could be
modified so that the pattern printedat the top of files already
contains %xEx so that logback-decoder can skip post compile processing
altogether.


Serialization to JSON or some other format is a relatively easy problem. 
I propose that we ignore serialization for the moment. The hard part is 
decoding log lines into event objects.

Please have a look at the FieldCapturer interface before reading on.
The link is http://tinyurl.com/fieldCapt

Due to the limitations of the pattern compiler, actual instances of
FieldCapturer would need to belong to a type extending Converter. This
is a little ugly since field capturers would not be doing any
conversions. We can tackle this marginal issue later.

Let us assume for the sake of this discussion that each event fits in
a single log line. Under this (obviously inaccurate) assumption,
decoding boils down to something like:

/**
  * Decode an log line as an ILoggingEvent.
  *
  * @param head - the first fieldCapturer returned as a result
  * of pattern parsing (see above)
  * @param inputLog the log line to decode
  *
  * @returnd the decoded ILoggingEvent
*/
public ILoggingEvent decode(FieldCapturer head, String inputLine) {

   FieldCapturer fieldCapturer = head;

   // ---------- build the pattern string -----------------
   StringBuilder sb = new StringBuilder();

   while(fieldCapturer != null) {
     String partialRegex = fieldCapturer.getRegexPattern();
     if(fieldCapturer.isCapturing())
       sb.append("(").append(partialRegex).append(")");
     else
       sb.append(partialRegex);
     }
     fieldCapturer = fieldCapturer.next();
   }

   // -------- do regex matching and capture fields --------
   String regex = sb.toString();
   Pattern pattern = Pattern.compile(regex);

   LoggingEvent event = new LoggingEvent();
   Matcher matcher = pattern.matcher(inputLine);

   if(matcher.matches()) {

     int i = 0;
     while(fieldCapturer != null) {
       if(fieldCapturer.isCapturing) {
         String fieldAsStr = matcher.group(i);
         i++;
         // much of the work is done here
         fieldCapturer.captureField(event, fieldAsStr);
       } else {
         // ignore literal text
       }
       fieldCapturer = fieldCapturer.next();
     }

   } else {
    logger.warn("Could not decode input line ["+inputLine+"]");
   }

   return event;
}


Much of the work is delegated to the field capturer chain. Obviously,
the decoder shown above assumes log events fitting in a single line. I
just wanted to share this design before addressing the more complicated
problem of multi-line decoding.

Comments most welcome.

-
Ceki
http://twitter.com/#!/ceki