[logback-user] logback autoscan is not working intermittently in k8s pod
Raymond Augé
raymond.auge at liferay.com
Fri Aug 20 16:05:22 CEST 2021
Here's a direct link [1] to the explanation of the issue which I confirmed
myself to be the actual issue I was facing.
Maybe it will help both the OP and logback :)
[1] https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/
On Fri, Aug 20, 2021 at 10:01 AM Raymond Augé <raymond.auge at liferay.com>
wrote:
> Hi, a long time lurker here 🙂.
>
> So, I suspect the issue is with k8s and it's volumes for config
> maps/secrets symlinks are a pain point in Java. Even inotify based
> WachService cannot handle them.
>
> The issue is that logback is only using a timestamp to detect
> configuration file changes, which is fine as we know in 99% of Java usages.
>
> However, this doesn't seem to work with the config map volumes supplied in
> a k8s env because they use symlinks which don't cause the timestamp to
> change, I agree it's weird and unexpected.
>
> However, supplying configuration files via K8s config maps is going to
> start being a much more common use case as time passes.
>
> Logback should probably start including a checksum to detect if the
> contents of the file have changed and use a logical OR with the timestamp
> check so that either changes causes a reload.
>
> FYI, I had to patch a similar issue in Apache Felix Fileinstall due to the
> same k8s config maps/secrets issue [1]
>
> My 2 cents,
> Ray
>
> [1] https://issues.apache.org/jira/browse/FELIX-6229
>
>
> On Fri., Aug. 20, 2021, 5:35 a.m. Ceki Gülcü, <ceki at qos.ch> wrote:
>
>> Hello Kedar,
>>
>> Have you tried to reproduce the problem on a stand alone machine without
>> k8?
>>
>>
>>
>> On 20/08/2021 00:41, Kedar Sirshikar wrote:
>> > Hello Ceki,
>> > I tried debugging by reproducing the issue and observed that after 7th
>> > attempt of log change, autoScan stops working.
>> >
>> > Have you came across similar issue in 1.2.3 version where autoScan does
>> > not pick changes from config xml if it gets changed after a certain
>> count ?
>> > In the meantime, I am trying to see if same issue occurs on 1.1.3
>> > version because setMainWatchURL method inConfigurationWatchListUtil is
>> > different on both versions.
>> >
>> >
>> > Thanks,
>> > Kedar.
>> >
>> >
>> > On Sat, Aug 14, 2021 at 3:00 AM <logback-user-request at qos.ch
>> > <mailto:logback-user-request at qos.ch>> wrote:
>> >
>> > Send logback-user mailing list submissions to
>> > logback-user at qos.ch <mailto:logback-user at qos.ch>
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> > http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
>> > or, via email, send a message with subject or body 'help' to
>> > logback-user-request at qos.ch <mailto:logback-user-request at qos.ch>
>> >
>> > You can reach the person managing the list at
>> > logback-user-owner at qos.ch <mailto:logback-user-owner at qos.ch>
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of logback-user digest..."
>> >
>> >
>> > Today's Topics:
>> >
>> > 1. Re: logback autoscan is not working intermittently in k8s pod
>> > (Ceki Gülcü)
>> >
>> >
>> >
>> ----------------------------------------------------------------------
>> >
>> > Message: 1
>> > Date: Fri, 13 Aug 2021 14:51:09 +0200
>> > From: Ceki Gülcü <ceki at qos.ch <mailto:ceki at qos.ch>>
>> > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > Subject: Re: [logback-user] logback autoscan is not working
>> > intermittently in k8s pod
>> > Message-ID: <f39cbede-6e95-7e60-d01e-ca27599bf453 at qos.ch
>> > <mailto:f39cbede-6e95-7e60-d01e-ca27599bf453 at qos.ch>>
>> > Content-Type: text/plain; charset=utf-8; format=flowed
>> >
>> > Hello Kedar,
>> >
>> >
>> > I see that changes to the logback config file are picked up. There
>> are
>> > some errors reported when processing the configuration file but
>> these
>> > are unrelated to the issue at hand, i.e. auto scan not working.
>> >
>> > I don't have further suggestions.
>> >
>> > --Ceki
>> >
>> >
>> > On 13/08/2021 09:59, Kedar Sirshikar wrote:
>> > > Hello team,
>> > > Just trying to check if anyone got a chance to look at this case?
>> > > Appreciate if any clue/hint is provided to resolve this issue.
>> > >
>> > > Thanks,
>> > > Kedar.
>> > >
>> > >
>> > > On Wed, Aug 11, 2021 at 12:57 PM Kedar Sirshikar
>> > > <kedargsirshikar at gmail.com <mailto:kedargsirshikar at gmail.com>
>> > <mailto:kedargsirshikar at gmail.com
>> > <mailto:kedargsirshikar at gmail.com>>> wrote:
>> > >
>> > > Resending the attachment by updating the subject. Please let
>> > me know
>> > > if any additional details are required from my side.
>> > >
>> > > Thanks,
>> > > Kedar.
>> > >
>> > >
>> > > ---------- Forwarded message ---------
>> > > From: *Kedar Sirshikar* <kedargsirshikar at gmail.com
>> > <mailto:kedargsirshikar at gmail.com>
>> > > <mailto:kedargsirshikar at gmail.com
>> > <mailto:kedargsirshikar at gmail.com>>>
>> > > Date: Wed, Aug 11, 2021 at 7:58 AM
>> > > Subject: Re: logback-user Digest, Vol 115, Issue 2
>> > > To: <logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>>
>> > >
>> > >
>> > > Hi Ceki, Thank you for the reply. Forgot to send the logs
>> > earlier so
>> > > please refer attachment.
>> > > I have also added Notes at lines 1 and 91 to describe the
>> test.
>> > >
>> > >
>> > > Thanks,
>> > > Kedar.
>> > >
>> > >
>> > > On Wed, Aug 11, 2021 at 3:00 AM <logback-user-request at qos.ch
>> > <mailto:logback-user-request at qos.ch>
>> > > <mailto:logback-user-request at qos.ch
>> > <mailto:logback-user-request at qos.ch>>> wrote:
>> > >
>> > > Send logback-user mailing list submissions to
>> > > logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
>> > >
>> > > To subscribe or unsubscribe via the World Wide Web, visit
>> > > http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
>> > > <http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>>
>> > > or, via email, send a message with subject or body
>> 'help' to
>> > > logback-user-request at qos.ch <mailto:logback-user-request at qos.ch>
>> > <mailto:logback-user-request at qos.ch
>> > <mailto:logback-user-request at qos.ch>>
>> > >
>> > > You can reach the person managing the list at
>> > > logback-user-owner at qos.ch <mailto:logback-user-owner at qos.ch>
>> > <mailto:logback-user-owner at qos.ch <mailto:logback-user-owner at qos.ch
>> >>
>> > >
>> > > When replying, please edit your Subject line so it is
>> > more specific
>> > > than "Re: Contents of logback-user digest..."
>> > >
>> > >
>> > > Today's Topics:
>> > >
>> > > 1. logback autoscan is not working intermittently in
>> > k8s pod
>> > > (Kedar Sirshikar)
>> > > 2. Re: logback autoscan is not working intermittently
>> > in k8s pod
>> > > (Ceki Gülcü)
>> > >
>> > >
>> > >
>> >
>> ----------------------------------------------------------------------
>> > >
>> > > Message: 1
>> > > Date: Tue, 10 Aug 2021 23:07:51 -0700
>> > > From: Kedar Sirshikar <kedargsirshikar at gmail.com
>> > <mailto:kedargsirshikar at gmail.com>
>> > > <mailto:kedargsirshikar at gmail.com
>> > <mailto:kedargsirshikar at gmail.com>>>
>> > > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
>> > > Subject: [logback-user] logback autoscan is not working
>> > > intermittently
>> > > in k8s pod
>> > > Message-ID:
>> > >
>> > >
>> > <CAO=
>> cJL+EMFCTe7En+7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com <mailto:
>> cJL%2BEMFCTe7En%2B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com>
>> > >
>> > <mailto:
>> cJL%2BEMFCTe7En%2B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com
>> <mailto:
>> cJL%252BEMFCTe7En%252B7nqUmzaxQoYVEYN1GXCYwdwfkt0KbjiDQ at mail.gmail.com>>>
>> > > Content-Type: text/plain; charset="utf-8"
>> > >
>> > > Hi team,
>> > > I have raised a detailed query here
>> > >
>> > <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> >
>> > >
>> > <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> >>>
>> > > about
>> > > the issue I am facing.
>> > > Can someone please share inputs or any hint for solving
>> the
>> > > above problem?
>> > > Let me know if any details are required from my side.
>> > >
>> > > Thanks,
>> > > Kedar.
>> > > -------------- next part --------------
>> > > An HTML attachment was scrubbed...
>> > > URL:
>> > >
>> > <
>> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
>> <
>> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
>> >
>> > >
>> > <
>> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
>> <
>> http://mailman.qos.ch/pipermail/logback-user/attachments/20210810/7ee3b6b7/attachment-0001.html
>> >>>
>> > >
>> > > ------------------------------
>> > >
>> > > Message: 2
>> > > Date: Wed, 11 Aug 2021 08:39:46 +0200
>> > > From: Ceki Gülcü <ceki at qos.ch <mailto:ceki at qos.ch>
>> > <mailto:ceki at qos.ch <mailto:ceki at qos.ch>>>
>> > > To: logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
>> > > Subject: Re: [logback-user] logback autoscan is not
>> working
>> > > intermittently in k8s pod
>> > > Message-ID: <2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch
>> > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch>
>> > > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch
>> > <mailto:2a210996-a364-e4f0-fe85-508c2af0b733 at qos.ch>>>
>> > > Content-Type: text/plain; charset=utf-8; format=flowed
>> > >
>> > >
>> > > Hello Kedar,
>> > >
>> > > Can you also provide the internal status messages
>> > generated by
>> > > logback?
>> > > --
>> > > Ceki
>> > >
>> > > On 11/08/2021 08:07, Kedar Sirshikar wrote:
>> > > > Hi team,
>> > > > I have raised a detailed query here
>> > > >
>> > >
>> > <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> >
>> > >
>> > <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> <
>> https://stackoverflow.com/questions/68721761/logback-autoscan-is-not-working-intermittently-in-k8s-pod
>> >>> about
>> > >
>> > > > the issue I am facing.
>> > > > Can someone please share inputs or any hint for
>> > solving the
>> > > above problem?
>> > > > Let me know if any details are required from my side.
>> > > >
>> > > > Thanks,
>> > > > Kedar.
>> > > >
>> > > > _______________________________________________
>> > > > logback-user mailing list
>> > > > logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > <mailto:logback-user at qos.ch <mailto:logback-user at qos.ch>>
>> > > > http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
>> > > <http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>>
>> > > >
>> > >
>> > >
>> > > ------------------------------
>> > >
>> >
>> >
>> > ------------------------------
>> >
>> > Subject: Digest Footer
>> >
>> > _______________________________________________
>> > logback-user mailing list
>> > logback-user at qos.ch <mailto:logback-user at qos.ch>
>> > http://mailman.qos.ch/mailman/listinfo/logback-user
>> > <http://mailman.qos.ch/mailman/listinfo/logback-user>
>> >
>> > ------------------------------
>> >
>> > End of logback-user Digest, Vol 115, Issue 6
>> > ********************************************
>> >
>> >
>> > _______________________________________________
>> > logback-user mailing list
>> > logback-user at qos.ch
>> > http://mailman.qos.ch/mailman/listinfo/logback-user
>> >
>> _______________________________________________
>> logback-user mailing list
>> logback-user at qos.ch
>> http://mailman.qos.ch/mailman/listinfo/logback-user
>
>
--
*Raymond Augé* (@rotty3000)
Senior Software Architect *Liferay, Inc.* (@Liferay)
OSGi Fellow, Java Champion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.qos.ch/pipermail/logback-user/attachments/20210820/fcb5073f/attachment-0001.html>
More information about the logback-user
mailing list