[logback-user] logback autoscan is not working intermittently in k8s pod
Kedar Sirshikar
kedargsirshikar at gmail.com
Tue Aug 24 03:47:13 CEST 2021
Thank you Raymond for sharing the inputs so looks like this is a bottleneck
from logback especially with k8s ConfigMap.
Is there any workaround (apart from restarting the java process) to
address this issue ? OR I'll have to wait until it gets fixed from logback
by considering checksum per below suggestion.
Hello Ceki, please share your advice if you have any.
Thanks,
Kedar.
On Fri, Aug 20, 2021 at 7:06 AM <logback-user-request at qos.ch> wrote:
> Send logback-user mailing list submissions to
> logback-user at qos.ch
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.qos.ch/mailman/listinfo/logback-user
> or, via email, send a message with subject or body 'help' to
> logback-user-request at qos.ch
>
> You can reach the person managing the list at
> logback-user-owner at qos.ch
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of logback-user digest..."
>
>
> Today's Topics:
>
> 1. Re: logback autoscan is not working intermittently in k8s pod
> (Raymond Augé)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 20 Aug 2021 10:05:22 -0400
> From: Raymond Augé <raymond.auge at liferay.com>
> To: logback users list <logback-user at qos.ch>
> Subject: Re: [logback-user] logback autoscan is not working
> intermittently in k8s pod
> Message-ID:
> <
> CAMm6HcC+fKzu4ux-z-WpDQK+LgWt_shwaEvB7OtR3J5dJe-UVQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Here's a direct link [1] to the explanation of the issue which I confirmed
> myself to be the actual issue I was facing.
>
> Maybe it will help both the OP and logback :)
>
> [1] https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
>
>
> On Fri, Aug 20, 2021 at 10:01 AM Raymond Augé <raymond.auge at liferay.com>
> wrote:
>
> > Hi, a long time lurker here 🙂.
> >
> > So, I suspect the issue is with k8s and it's volumes for config
> > maps/secrets symlinks are a pain point in Java. Even inotify based
> > WachService cannot handle them.
> >
> > The issue is that logback is only using a timestamp to detect
> > configuration file changes, which is fine as we know in 99% of Java
> usages.
> >
> > However, this doesn't seem to work with the config map volumes supplied
> in
> > a k8s env because they use symlinks which don't cause the timestamp to
> > change, I agree it's weird and unexpected.
> >
> > However, supplying configuration files via K8s config maps is going to
> > start being a much more common use case as time passes.
> >
> > Logback should probably start including a checksum to detect if the
> > contents of the file have changed and use a logical OR with the timestamp
> > check so that either changes causes a reload.
> >
> > FYI, I had to patch a similar issue in Apache Felix Fileinstall due to
> the
> > same k8s config maps/secrets issue [1]
> >
> > My 2 cents,
> > Ray
> >
> > [1] https://issues.apache.org/jira/browse/FELIX-6229
> >
> >
> > On Fri., Aug. 20, 2021, 5:35 a.m. Ceki Gülcü, <ceki at qos.ch> wrote:
> >
> >> Hello Kedar,
> >>
> >> Have you tried to reproduce the problem on a stand alone machine without
> >> k8?
> >>
> >>
> >>
> >> On 20/08/2021 00:41, Kedar Sirshikar wrote:
> >> > Hello Ceki,
> >> > I tried debugging by reproducing the issue and observed that after 7th
> >> > attempt of log change, autoScan stops working.
> >> >
> >> > Have you came across similar issue in 1.2.3 version where autoScan
> does
> >> > not pick changes from config xml if it gets changed after a certain
> >> count ?
> >> > In the meantime, I am trying to see if same issue occurs on 1.1.3
> >> > version because setMainWatchURL method inConfigurationWatchListUtil is
> >> > different on both versions.
> >> >
> >> >
> >> > Thanks,
> >> > Kedar.
> >> >
> >> >
> >> > On Sat, Aug 14, 2021 at 3:00 AM <logback-user-request at qos.ch
> >> > <mailto:logback-user-request at qos.ch>> wrote:
> >> >
> >> > Send logback-user mailing list submissions to
> >> > logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> >
> >> > To subscribe or unsubscribe via the World Wide Web, visit
> >> > http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
> >> > or, via email, send a message with subject or body 'help' to
> >> > logback-user-request at qos.ch <mailto:logback-user-request at qos.ch>
> >> >
> >> > You can reach the person managing the list at
> >> > logback-user-owner at qos.ch <mailto:logback-user-owner at qos.ch>
> >> >
> >> > When replying, please edit your Subject line so it is more
> specific
> >> > than "Re: Contents of logback-user digest..."
> >> >
> >> >
> >> > Today's Topics:
> >> >
> >> > 1. Re: logback autoscan is not working intermittently in k8s
> pod
> >> > (Ceki Gülcü)
> >> >
> >> >
> >> >
> >> ----------------------------------------------------------------------
> >> >
> >> > Message: 1
> >> > Date: Fri, 13 Aug 2021 14:51:09 +0200
> >> > From: Ceki Gülcü <ceki at qos.ch <mailto:ceki at qos.ch>>
> >> > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > Subject: Re: [logback-user] logback autoscan is not working
> >> > intermittently in k8s pod
> >> > Message-ID: <f39cbede-6e95-7e60-d01e-ca27599bf453 at qos.ch
> >> > <mailto:f39cbede-6e95-7e60-d01e-ca27599bf453 at qos.ch>>
> >> > Content-Type: text/plain; charset=utf-8; format=flowed
> >> >
> >> > Hello Kedar,
> >> >
> >> >
> >> > I see that changes to the logback config file are picked up. There
> >> are
> >> > some errors reported when processing the configuration file but
> >> these
> >> > are unrelated to the issue at hand, i.e. auto scan not working.
> >> >
> >> > I don't have further suggestions.
> >> >
> >> > --Ceki
> >> >
> >> >
> >> > On 13/08/2021 09:59, Kedar Sirshikar wrote:
> >> > > Hello team,
> >> > > Just trying to check if anyone got a chance to look at this
> case?
> >> > > Appreciate if any clue/hint is provided to resolve this issue.
> >> > >
> >> > > Thanks,
> >> > > Kedar.
> >> > >
> >> > >
> >> > > On Wed, Aug 11, 2021 at 12:57 PM Kedar Sirshikar
> >> > > <kedargsirshikar at gmail.com <mailto:kedargsirshikar at gmail.com>
> >> > <mailto:kedargsirshikar at gmail.com
> >> > <mailto:kedargsirshikar at gmail.com>>> wrote:
> >> > >
> >> > > Resending the attachment by updating the subject. Please
> let
> >> > me know
> >> > > if any additional details are required from my side.
> >> > >
> >> > > Thanks,
> >> > > Kedar.
> >> > >
> >> > >
> >> > > ---------- Forwarded message ---------
> >> > > From: *Kedar Sirshikar* <kedargsirshikar at gmail.com
> >> > <mailto:kedargsirshikar at gmail.com>
> >> > > <mailto:kedargsirshikar at gmail.com
> >> > <mailto:kedargsirshikar at gmail.com>>>
> >> > > Date: Wed, Aug 11, 2021 at 7:58 AM
> >> > > Subject: Re: logback-user Digest, Vol 115, Issue 2
> >> > > To: <logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>>
> >> > >
> >> > >
> >> > > Hi Ceki, Thank you for the reply. Forgot to send the logs
> >> > earlier so
> >> > > please refer attachment.
> >> > > I have also added Notes at lines 1 and 91 to describe the
> >> test.
> >> > >
> >> > >
> >> > > Thanks,
> >> > > Kedar.
> >> > >
> >> > >
> >> > > On Wed, Aug 11, 2021 at 3:00 AM <
> logback-user-request at qos.ch
> >> > <mailto:logback-user-request at qos.ch>
> >> > > <mailto:logback-user-request at qos.ch
> >> > <mailto:logback-user-request at qos.ch>>> wrote:
> >> > >
> >> > > Send logback-user mailing list submissions to
> >> > > logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
> >> > >
> >> > > To subscribe or unsubscribe via the World Wide Web,
> visit
> >> > > http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
> >> > > <http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>>
> >> > > or, via email, send a message with subject or body
> >> 'help' to
> >> > > logback-user-request at qos.ch <mailto:
> logback-user-request at qos.ch>
> >> > <mailto:logback-user-request at qos.ch
> >> > <mailto:logback-user-request at qos.ch>>
> >> > >
> >> > > You can reach the person managing the list at
> >> > > logback-user-owner at qos.ch <mailto:logback-user-owner at qos.ch>
> >> > <mailto:logback-user-owner at qos.ch <mailto:
> logback-user-owner at qos.ch
> >> >>
> >> > >
> >> > > When replying, please edit your Subject line so it is
> >> > more specific
> >> > > than "Re: Contents of logback-user digest..."
> >> > >
> >> > >
> >> > > Today's Topics:
> >> > >
> >> > > 1. logback autoscan is not working intermittently
> in
> >> > k8s pod
> >> > > (Kedar Sirshikar)
> >> > > 2. Re: logback autoscan is not working
> intermittently
> >> > in k8s pod
> >> > > (Ceki Gülcü)
> >> > >
> >> > >
> >> > >
> >> >
> >> ----------------------------------------------------------------------
> >> > >
> >> > > Message: 1
> >> > > Date: Tue, 10 Aug 2021 23:07:51 -0700
> >> > > From: Kedar Sirshikar <kedargsirshikar at gmail.com
> >> > <mailto:kedargsirshikar at gmail.com>
> >> > > <mailto:kedargsirshikar at gmail.com
> >> > <mailto:kedargsirshikar at gmail.com>>>
> >> > > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
> >> > > Subject: [logback-user] logback autoscan is not working
> >> > > intermittently
> >> > > in k8s pod
> >> > > Message-ID:
> >> > >
> >> > >
> >> > <CAO=
> >> cJL+EMFCTe7En+7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com <mailto:
> >> cJL%2BEMFCTe7En%2B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com>
> >> > >
> >> > <mailto:
> >> cJL%2BEMFCTe7En%2B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com
> >> <mailto:
> >> cJL%252BEMFCTe7En%252B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com
> >>>
> >> > > Content-Type: text/plain; charset="utf-8"
> >> > >
> >> > > Hi team,
> >> > > I have raised a detailed query here
> >> > >
> >> > <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> >
> >> > >
> >> > <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> >>>
> >> > > about
> >> > > the issue I am facing.
> >> > > Can someone please share inputs or any hint for solving
> >> the
> >> > > above problem?
> >> > > Let me know if any details are required from my side.
> >> > >
> >> > > Thanks,
> >> > > Kedar.
> >> > > -------------- next part --------------
> >> > > An HTML attachment was scrubbed...
> >> > > URL:
> >> > >
> >> > <
> >>
> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
> >> <
> >>
> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
> >> >
> >> > >
> >> > <
> >>
> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
> >> <
> >>
> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
> >> >>>
> >> > >
> >> > > ------------------------------
> >> > >
> >> > > Message: 2
> >> > > Date: Wed, 11 Aug 2021 08:39:46 +0200
> >> > > From: Ceki Gülcü <ceki at qos.ch <mailto:ceki at qos.ch>
> >> > <mailto:ceki at qos.ch <mailto:ceki at qos.ch>>>
> >> > > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
> >> > > Subject: Re: [logback-user] logback autoscan is not
> >> working
> >> > > intermittently in k8s pod
> >> > > Message-ID: <
> 2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch
> >> > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch>
> >> > > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch
> >> > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch>>>
> >> > > Content-Type: text/plain; charset=utf-8; format=flowed
> >> > >
> >> > >
> >> > > Hello Kedar,
> >> > >
> >> > > Can you also provide the internal status messages
> >> > generated by
> >> > > logback?
> >> > > --
> >> > > Ceki
> >> > >
> >> > > On 11/08/2021 08:07, Kedar Sirshikar wrote:
> >> > > > Hi team,
> >> > > > I have raised a detailed query here
> >> > > >
> >> > >
> >> > <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> >
> >> > >
> >> > <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> <
> >>
> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
> >> >>> about
> >> > >
> >> > > > the issue I am facing.
> >> > > > Can someone please share inputs or any hint for
> >> > solving the
> >> > > above problem?
> >> > > > Let me know if any details are required from my
> side.
> >> > > >
> >> > > > Thanks,
> >> > > > Kedar.
> >> > > >
> >> > > > _______________________________________________
> >> > > > logback-user mailing list
> >> > > > logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
> >> > > > http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
> >> > > <http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>>
> >> > > >
> >> > >
> >> > >
> >> > > ------------------------------
> >> > >
> >> >
> >> >
> >> > ------------------------------
> >> >
> >> > Subject: Digest Footer
> >> >
> >> > _______________________________________________
> >> > logback-user mailing list
> >> > logback-user at qos.ch <mailto:logback-user at qos.ch>
> >> > http://mailman.qos.ch/mailman/listinfo/logback-user
> >> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
> >> >
> >> > ------------------------------
> >> >
> >> > End of logback-user Digest, Vol 115, Issue 6
> >> > ********************************************
> >> >
> >> >
> >> > _______________________________________________
> >> > logback-user mailing list
> >> > logback-user at qos.ch
> >> > http://mailman.qos.ch/mailman/listinfo/logback-user
> >> >
> >> _______________________________________________
> >> logback-user mailing list
> >> logback-user at qos.ch
> >> http://mailman.qos.ch/mailman/listinfo/logback-user
> >
> >
>
> --
> *Raymond Augé* (@rotty3000)
> Senior Software Architect *Liferay, Inc.* (@Liferay)
> OSGi Fellow, Java Champion
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mailman.qos.ch/pipermail/logback-user/attachments/20210820/fcb5073f/attachment.html
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> logback-user mailing list
> logback-user at qos.ch
> http://mailman.qos.ch/mailman/listinfo/logback-user
>
> ------------------------------
>
> End of logback-user Digest, Vol 115, Issue 12
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20210823/4fb238c8/attachment-0001.html>
More information about the logback-user
mailing list