[logback-dev] Protobuf

[Maarten Bosteels]

On Wed, Mar 18, 2009 at 10:25 PM, Ceki Gulcu <ceki at qos.ch> wrote:


>
> Are you using the standard corpus with 50'000? If so, 12MB for 50'000
> elements translates to about 240 bytes per message which is pretty good. The
> CPU times, e.g. 1883 ms, are fantastic.


yep,

ILoggingEvent[] eventArray = Corpus.makeStandardCorpus();

    for (int i=0; i<10; i++) {
      ObjectOutputStream oos = new ObjectOutputStream(new
FileOutputStream("/tmp/logback-java.ser." + i));
      OutputStream os = new BufferedOutputStream(new
FileOutputStream("/tmp/logback-protobuf.ser" + i));
      GZIPOutputStream gos = new GZIPOutputStream(new
FileOutputStream("/tmp/logback-proto." + i + ".gz"), 8000);

      DeflaterOutputStream dos = new DeflaterOutputStream(
          new FileOutputStream("/tmp/logback-protobuf.def." + i),
          new Deflater(Deflater.BEST_SPEED));

      serialize(eventArray, oos);
      serializeWithProtobuf("normal", eventArray, os);
      serializeWithProtobuf("gzip  ", eventArray, gos);
      serializeWithProtobuf("def   ", eventArray, dos);
      oos.close();
      os.close();
      gos.close();
    }

Still have to create a multi-threaded test, but I suspect it won't make much
difference.

Maarten


>
> Maarten Bosteels wrote:
>
>> Hi Ceki,
>>
>> Keep in mind that I still have to double check my LoggingEvent-to-protobuf
>> converter.
>>
>> Serializing the corpus with different formats:
>> java serialization   => 16 MB  (16108602)
>> protobuf                => 33 MB  (34385867 bytes)
>> protobuf + gzip      => 9.9 MB  (10354646 bytes)
>> protbuf + deflater   =>  12 MB  (12006206 bytes)
>>
>> speed:
>>
>> java serialization:    4330 ms
>> protobuf                   928 ms
>> protobuf + gzip       3146 ms protobuf + deflater  1883 ms
>>
>> So I would choose for protobuf + deflater :  25% smaller than java
>> serilaization and twice as fast.
>>
>> Deflater means using  java.util.zip.Deflater.BEST_SPEED
>>
>> regards,
>> Maarten
>>
>>
>> On Wed, Mar 18, 2009 at 9:49 PM, Ceki Gulcu <ceki at qos.ch> wrote:
>>
>>
>>    Hello Maarten,
>>
>>    It's interesting to learn that protobuf is faster than plain old
>>    java serialization. Thanks.
>>
>>    How about the size of the stored data? How many bytes does a logging
>>    event use on average?
>>
>>    Maarten Bosteels wrote:
>>
>>        Hi,
>>
>>        No objections at all.
>>
>>        Looking at my .proto file it's indeed pretty obvious that
>>        CallerData and StackTraceElement are quite similar :-)
>>
>> http://code.google.com/p/firewood/source/browse/trunk/compare-formats/src/main/java/com/googlecode/firewood/protobuf/logging.proto
>>
>>        I've done some benchmarking with protobuf (using the Corpus),
>>        and it seems to be at least 4 times faster than Java serialization.
>>        Will soon post the results and the code.
>>
>>        regards,
>>        Maarten
>>
>>
>>    --    Ceki Gülcü
>>    Logback: The reliable, generic, fast and flexible logging framework
>>    for Java.
>>    http://logback.qos.ch
>>    _______________________________________________
>>    logback-dev mailing list
>>    logback-dev at qos.ch
>>    http://qos.ch/mailman/listinfo/logback-dev
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> logback-dev mailing list
>> logback-dev at qos.ch
>> http://qos.ch/mailman/listinfo/logback-dev
>>
>
> --
> Ceki Gülcü
> Logback: The reliable, generic, fast and flexible logging framework for
> Java.
> http://logback.qos.ch
> _______________________________________________
> logback-dev mailing list
> logback-dev at qos.ch
> http://qos.ch/mailman/listinfo/logback-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://qos.ch/pipermail/logback-dev/attachments/20090318/f0705653/attachment-0001.htm>