[logback-dev] [JIRA] Commented: (LBCORE-128) Please support implementation of binary log files in RollingFileAppender/FileAppender

[Joern Huxhorn (JIRA)]

    [ http://jira.qos.ch/browse/LBCORE-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=11569#action_11569 ] 

Joern Huxhorn commented on LBCORE-128:
--------------------------------------

That's right, the file header does only contain very minimal info.

- a magic value (int) that's used to identify my filebuffers in general (0x0B501E7E => obsolete)
- another magic value (int) that's used to identify Lilith files in particular (0x0B5E55ED => obsessed)
- the size of the following metadata (int)
- protobuf encoded metadata that's retrieved/set as a Map<String,String>
  this metadata contains info about:
  - the compression used, if any
  - the content type (logging or access)
  - primary and secondary identifier that can be used to identify the source of the events. The secondary identifier is usually a timestamp of creation.

The offset of the first event is therefore 12+(sizeof metadata).
Each event, again, is an int for the size followed by that amount of bytes of event data.

Very simple but not suitable for replace or overwrite.
Lilith is also using an additional index file which stores the offset for every event so it's possible to query event n very fast. This doesn't have to be created by the appender, though, since it can already be generated by Lilith for a given file.

Why would the start position be lost? This can only happen if an event is written only partially. In that case I'd start with a new file (as a recovery). Such a marker could be added quite easily but I didn't do it so far because it simply wasn't necessary, yet.

Hope this info helps a bit.

> Please support implementation of binary log files in RollingFileAppender/FileAppender
> -------------------------------------------------------------------------------------
>
>                 Key: LBCORE-128
>                 URL: http://jira.qos.ch/browse/LBCORE-128
>             Project: logback-core
>          Issue Type: Improvement
>          Components: Appender
>    Affects Versions: 0.9.17
>            Reporter: Joern Huxhorn
>            Assignee: Ceki Gulcu
>
> This was discussed briefly at http://marc.info/?l=logback-dev&m=124905434331308&w=2 and I forgot to file a ticket about this.
> Currently, RandomFileAppender => FileAppender => WriterAppender is using the following method in WriterAppender to actually write the data:
> protected void writerWrite(String s, boolean flush) throws IOException
> Please add an additional method like
> protected void writerWrite(byte[] bytes, boolean flush) throws IOException
> to write to the underlying stream directly.
> writerWrite(String, boolean) could call that method after performing the transformation internally, making this change transparent for the rest of the implementation.
> Using a binary format for logfiles could have tremendous performance impact as can be seen here: http://sourceforge.net/apps/trac/lilith/wiki/SerializationPerformance

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira