<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and (-webkit-min-device-pixel-ratio: 2) {
#email-body { min-width: 30em !important; }
#email-page { padding: 8px !important; }
#email-banner { padding: 8px 8px 0 8px !important; }
#email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
#email-fields { padding: 0 8px 8px 8px !important; }
#email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#f0f0f0;color:#000000;width:100%;">
<tr valign="top">
<td id="email-page" style="padding:16px !important;">
<table align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;">
<tr valign="top">
<td bgcolor="#003366" style="background-color:#003366;color:#ffffff;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img src="http://jira.qos.ch/s/en_USet69y6/731/19/_/jira-logo-scaled.png" alt="" style="vertical-align:top;" /></td>
</tr><tr valign="top">
<td id="email-banner" style="padding:32px 32px 0 32px;">
<table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="width:100%;">
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
<img id="email-avatar" src="http://jira.qos.ch/secure/useravatar?avatarId=10122" alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px 16px 0;" />
<div id="email-action" style="padding: 0 0 8px 0;font-size:12px;line-height:18px;">
<a class="user-hover" rel="gunnar" id="email_gunnar" href="http://jira.qos.ch/secure/ViewProfile.jspa?name=gunnar" style="color:#326ca6;">Gunnar Wagenknecht</a>
commented on <img src="http://jira.qos.ch/images/icons/bug.gif" height="16" width="16" border="0" align="absmiddle" alt="Bug"> <a style='color:#326ca6;text-decoration:none;' href='http://jira.qos.ch/browse/LBCLASSIC-331'>LBCLASSIC-331</a>
</div>
<div id="email-summary" style="font-size:16px;line-height:20px;padding:2px 0 16px 0;">
<a style='color:#326ca6;text-decoration:none;' href='http://jira.qos.ch/browse/LBCLASSIC-331'><strong>Console appender may deadlock when System.out is hijacked</strong></a>
</div>
</td>
</tr>
</table>
</td>
</tr>
<tr valign="top">
<td id="email-fields" style="padding:0 32px 32px 32px;">
<table border="0" cellpadding="0" cellspacing="0" style="padding:0;text-align:left;width:100%;" width="100%">
<tr valign="top">
<td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
<td>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr valign="top">
<td colspan="2" style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 16px 0;width:100%;">
<div class="comment-block" style="background-color:#edf5ff;border:1px solid #dddddd;color:#000000;padding:12px;">I think it's reproducible when setting the level to DEBUG again.
<br/>
<br/>
I looked a bit further and here are my findings:
<br/>
- Thread-21 has a lock on Object (id=202). This must be the lock object within Logback ConsoleAppender because all other threads are waiting on this one (as is NioProcessor-2 for example).
<br/>
- Thread-21 is waiting for Window (id=485) but not because of a lock but because of an Object.wait() call on Window.
<br/>
- Karaf must have replaced System.out with a ConsoleOutputStream that wraps a ChannelOutputStream from SSHD.
<br/>
- ChannelOutputStream.flush() is called recursively in Thread-21 because of the console logging.
<br/>
- The formatting above is poor. In thread NioProcessor-2 it should read "waiting for Object (id=202) which is owned by [Thread-21] which is waiting on Window (id=485)";
<br/>
<br/>
The situation must be as follows: Thread-21 is waiting for some other thread to free space in the Window object (waitForSpace). This is implemented using Object.wait/notify(). However, no other thread will ever call Window.notify because they are all stuck in ConsoleAppender while doing some log output because the ConsoleAppender lock is held by Thread-21 which is sleeping.
<br/>
<br/>
I think the issue is a combination of SSHD + Karaf + Logback. Logback is configured to write debug info to System.out Karaf hijacks System.out to properly route output for SSH console sessions established via SSHD. SSHD uses a wait/notify pattern when flushing content from the server to clients via an asynchronous threading model. However, the flushing ("notifying") thread is stuck because of Logback OutputStreamAppender synchronization behavior which only allows one thread at a time to write to any output stream.
<br/>
<br/>
So, maybe a possible solution is to disable the synchronization for ConsoleAppender only?</div>
<div style="color:#505050;padding:4px 0 0 0;"> </div>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td><!-- End #email-page -->
</tr>
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: 0 16px 16px 16px;text-align:center;">
This message is automatically generated by JIRA.<br />
If you think it was sent incorrectly, please contact your <a style='color:#326ca6;' href='http://jira.qos.ch/secure/ContactAdministrators!default.jspa'>JIRA administrators</a>.<br />
For more information on JIRA, see: <a style='color:#326ca6;' href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
</td>
</tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->