[logback-user] logback and flume

Wed Mar 2 14:54:19 UTC 2016

Hi,
Thanks, that’s useful to know.
Rds,
Mike

-----Original Message-----
From: logback-user [mailto:logback-user-bounces at qos.ch] On Behalf Of Adam Gent
Sent: 02 March 2016 14:52
To: logback users list
Subject: Re: [logback-user] logback and flume

For what its worth (hopefully relevant, helpful and not too tangental) I found it more reliable to publish to a RabbitMQ cluster (logback/log4j plugin) and then have Flume consume from Rabbit which then pushes to Elastic Search.

I really wish I could give high praise to flume but it really has given us a ton of headaches. It will run out of memory, drop messages, and silently fail. This maybe because we just don't have it configured correctly and/or just that the Elastic Search plugin for flume is not mature enough. The Elastic Search flume plugin for sure has some bugs when it sees JSON like data in a message body it tries to parse it even if its not JSON.

Since flume was only used to read message from RabbitMQ and then push messages to Elastic we have a replacement in the works written in Rust but is not production ready.

RabbitMQ is actually pretty darn good logging aggregation system even if that is not what it is designed for.

On Wed, Mar 2, 2016 at 9:35 AM, Mike Lewis <MLewis at nephilaadvisors.co.uk> wrote:
> Hi,
>
> Sure, first of all here is an example flume.conf file:
>
> agent.sources=s1
> agent.channels=c1
> agent.sinks=k1
> agent.sources.s1.type=avro
> agent.sources.s1.channels=c1
> agent.sources.s1.bind=0.0.0.0
> agent.sources.s1.port=55555
> agent.channels.c1.type=memory
> agent.channels.c1.capacity = 10000
> agent.channels.c1.transactionCapacity=100
> agent.sinks.k1.type=logger
> agent.sinks.k1.channel=c1
>
>
> The source of my issues was the missing type=avro line.
>
>  I used Scala for my test application, the dependencies I needed were:
>
> libraryDependencies ++= Seq (
>         "ch.qos.logback" % "logback-classic" % "1.1.2",
>         "com.teambytes.logback" %% "logback-flume-appender" % "0.0.9",
>         "com.gilt.flume" % "logback-flume-appender" % "0.1.7"
> )
>
> The test application itself is trivial:
>
> import org.slf4j.{LoggerFactory, Logger}
>
>
> object LoggingApp extends App {
>
>   val logger : Logger =
>     LoggerFactory.getILoggerFactory.getLogger("com.xxxxx.xxxxx")
>
>   logger.debug("*Starting App.")
>
>   for (message <- 1 to 2) {
>     logger.info(s"*Processing: $message")
>   }
>
>   logger.debug("*Stopping App.")
> }
>
> I took the logback.xml from
> https://github.com/gilt/logback-flume-appender
> Note, there is a typo, the class needs to be "com.gilt.flume.logback.FlumeLogstashV1Appender"
>
> n.b. I'm not sure of the benefit of flume vs syslog-ng etc... I'm just
> trying to start looking at a  logback -> (flume|syslog-ng)->elastic search -> kibana stack for log file management.
>
> Regards,
> Mike
>
>
>
> -----Original Message-----
> From: logback-user [mailto:logback-user-bounces at qos.ch] On Behalf Of
> Ceki Gulcu
> Sent: 02 March 2016 14:29
> To: logback-user at qos.ch
> Subject: Re: [logback-user] logback and flume
>
> Hi Mike,
>
> Could you describe the steps you have undertaken for future reference?
>
> --
> Ceki
>
> On 3/2/2016 15:25, Mike Lewis wrote:
>> I managed to resolve this, I had not set the type=avro in my
>> flume.conf file.
>>
>> *From:*logback-user [mailto:logback-user-bounces at qos.ch] *On Behalf
>> Of *Mike Lewis
>> *Sent:* 02 March 2016 12:50
>> *To:* logback-user at qos.ch
>> *Subject:* [logback-user] logback and flume
>>
>> Hi,
>>
>> I was just wondering if anyone had experience using logback with flume?
>> If so,
>>
>> Could you let me know the particular libs you used, I’ve tried a few
>> things with
>>
>> no success.
>>
>>
>> Regards,
>>
>> Mike Lewis
>>
>>
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>> This email has been sent to you on behalf of Nephila Advisors LLC
>> (“Advisors”). Advisors provides consultancy services to Nephila
>> Capital Ltd. (“Capital”), an investment advisor managed and carrying
>> on business in Bermuda. Advisors and its employees do not act as
>> agents for Capital or the funds it advises and do not have the
>> authority to bind Capital or such funds to any transaction or agreement.
>>
>> The information in this e-mail, and any attachment therein, is
>> confidential and for use by the addressee only. Any use, disclosure,
>> reproduction, modification or distribution of the contents of this
>> e-mail, or any part thereof, other than by the intended recipient, is
>> strictly prohibited. If you are not the intended recipient, please
>> return the e-mail to the sender and delete it from your computer.
>> This email is for information purposes only, nothing contained herein
>> constitutes an offer to sell or buy securities, as such an offer may
>> only be made from a properly authorized offering document. Although
>> Nephila attempts to sweep e-mail and attachments for viruses, it does
>> not guarantee that either are virus-free and accepts no liability for
>> any damage sustained as a result of viruses.
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>>
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>> This email has been sent to you on behalf of Nephila Advisors UK
>> (“Advisors UK”). Advisors UK provides consultancy services to Nephila
>> Capital Ltd. (“Capital”), an investment advisor managed and carrying
>> on business in Bermuda. Advisors UK and its employees do not act as
>> agents for Capital or the funds it advises and do not have the
>> authority to bind Capital or such funds to any transaction or agreement.
>>
>> The information in this e-mail, and any attachment therein, is
>> confidential and for use by the addressee only. Any use, disclosure,
>> reproduction, modification or distribution of the contents of this
>> e-mail, or any part thereof, other than by the intended recipient, is
>> strictly prohibited. If you are not the intended recipient, please
>> return the e-mail to the sender and delete it from your computer.
>> This email is for information purposes only, nothing contained herein
>> constitutes an offer to sell or buy securities, as such an offer may
>> only be made from a properly authorized offering document. Although
>> Nephila attempts to sweep e-mail and attachments for viruses, it does
>> not guarantee that either are virus-free and accepts no liability for
>> any damage sustained as a result of viruses.
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>>
>>
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>> This email has been sent to you on behalf of Nephila Advisors LLC
>> (“Advisors”). Advisors provides consultancy services to Nephila
>> Capital Ltd. (“Capital”), an investment advisor managed and carrying
>> on business in Bermuda. Advisors and its employees do not act as
>> agents for Capital or the funds it advises and do not have the
>> authority to bind Capital or such funds to any transaction or agreement.
>>
>> The information in this e-mail, and any attachment therein, is
>> confidential and for use by the addressee only. Any use, disclosure,
>> reproduction, modification or distribution of the contents of this
>> e-mail, or any part thereof, other than by the intended recipient, is
>> strictly prohibited. If you are not the intended recipient, please
>> return the e-mail to the sender and delete it from your computer.
>> This email is for information purposes only, nothing contained herein
>> constitutes an offer to sell or buy securities, as such an offer may
>> only be made from a properly authorized offering document. Although
>> Nephila attempts to sweep e-mail and attachments for viruses, it does
>> not guarantee that either are virus-free and accepts no liability for
>> any damage sustained as a result of viruses.
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>>
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>> This email has been sent to you on behalf of Nephila Advisors UK
>> (“Advisors UK”). Advisors UK provides consultancy services to Nephila
>> Capital Ltd. (“Capital”), an investment advisor managed and carrying
>> on business in Bermuda. Advisors UK and its employees do not act as
>> agents for Capital or the funds it advises and do not have the
>> authority to bind Capital or such funds to any transaction or agreement.
>>
>> The information in this e-mail, and any attachment therein, is
>> confidential and for use by the addressee only. Any use, disclosure,
>> reproduction, modification or distribution of the contents of this
>> e-mail, or any part thereof, other than by the intended recipient, is
>> strictly prohibited. If you are not the intended recipient, please
>> return the e-mail to the sender and delete it from your computer.
>> This email is for information purposes only, nothing contained herein
>> constitutes an offer to sell or buy securities, as such an offer may
>> only be made from a properly authorized offering document. Although
>> Nephila attempts to sweep e-mail and attachments for viruses, it does
>> not guarantee that either are virus-free and accepts no liability for
>> any damage sustained as a result of viruses.
>> ---------------------------------------------------------------------
>> -
>> ----------------------------------------------------
>>
>>
>>
>> _______________________________________________
>> logback-user mailing list
>> logback-user at qos.ch
>> http://mailman.qos.ch/mailman/listinfo/logback-user
>>
> _______________________________________________
> logback-user mailing list
> logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user
>
> ----------------------------------------------------------------------
> ----------------------------------------------------
> This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.
>
> The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.
> ----------------------------------------------------------------------
> ----------------------------------------------------
>
> ----------------------------------------------------------------------
> ----------------------------------------------------
> This email has been sent to you on behalf of Nephila Advisors UK (“Advisors UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors UK and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.
>
> The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.
> ----------------------------------------------------------------------
> ----------------------------------------------------
> _______________________________________________
> logback-user mailing list
> logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user

--
CTO
SnapHop (snaphop.com)
(twitter) @agentgt (linkedin) http://www.linkedin.com/in/agentgt
(cell) 781-883-5182
_______________________________________________
logback-user mailing list
logback-user at qos.ch
http://mailman.qos.ch/mailman/listinfo/logback-user

--------------------------------------------------------------------------------------------------------------------------
This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.

The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.
--------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------------
This email has been sent to you on behalf of Nephila Advisors UK (“Advisors UK”). Advisors UK provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors UK and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement.

The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses.
--------------------------------------------------------------------------------------------------------------------------