[logback-user] Log events delivery guarantee.

[Ralph Goers]

> On Jun 2, 2017, at 2:58 PM, Oleksandr Gavenko <gavenkoa at gmail.com> wrote:
> 
> On Fri, Jun 2, 2017 at 10:53 PM, Ralph Goers <rgoers at apache.org> wrote:
>> It uses BerkelyDB and writes the event to it before passing the event to
>> another thread where it uses the Flume RPC client to send the event off. You
>> could copy just the BerkelyDB logic and tie it to another delivery mechanism
>> if you want.
> 
> Thanks for suggestion!
> 
> I didn't profile myself how long latency in writing single row to DB,
> numbers from Internet
> about 1 ms.
> 
> I think that in case of embedded DB latency would be much smaller.
> 
> According to:
> 
> http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html
> 
> BerkelyDB add 1 MiB overhead in distribution which is acceptable.
> Don't know about runtime overhead.
> 
> Embedded DB provides reliable solution for data consistency and allows
> keep it structured.
> 
> I am not sure if DB size stay low if you add/del data over period of
> time. I don't like to maintain this DB.
> 
> With some file rotation solution you can write to new file and process
> old one (and just delete it at the end).

As I recall the architecture of BerkelyDB is such that it uses fixed sized files and writes to them and swaps the old files out as they empty. Basically, you just use it and forget about it.  I have had one user complaint that it didn’t gracefully handle the case when the disk was full, but simply adding decent OS monitoring can avoid that.

In my testing I found that the performance of BerkelyDB wasn’t quite as good as Flume’s File channel, but Flume has a lot of dependencies and BerkelyDB has very few.  Both seem to be pretty reliable.

Ralph