Jenkins‎ > ‎

Jenkins-5413 Work-around

The Jenkins issue #5413 is about "SCM polling on slaves is hanging" .

What the problem boils down to is that many remote operations are performed synchronously causing the channel object to be locked while a response returns. In situations where a lengthy remote operations is using the channel, SCM polling can be blocked waiting for the monitor on the channel to be released. In extreme situations, all the polling threads can wind up waiting on object monitors for the channel objects, preventing further processing of polling tasks.

In the comments to the issue vjuranek suggests a work-around for this problem. It's a small groovy scripts that identifies the SCM polling threads and terminates them. The good thing is that it prevents you from restarting the Jenkins server over and over again. The not-so-good thing is that it might also terminate healthy poll threads. But if you schedule it to run frequently - but not too frequently, it will do the trick.

Please note that we are looking into a permanent fix to issue 5413, after which this work-around will not be necessary any more.

...In the meantime, this article is a formal step-by-step instruction to the work-around:

Preconditions:
You must have installed and configured the Groovy plugin properly on Jenkins and have the Groovy binaries on the machine, please see the documentation for the Groovy plugin for more info.

Setup:

1). Create a new free-style Jenkins job and give it a matching name like "poll_threads_maintenance"

2). Add a new Build step, of type "Execute system Groovy script"

3). Make sure you select "Groovy command"

4). Paste the following code into the text box.

Thread.getAllStackTraces().keySet().each() {
  item ->
  if (item.getName().contains("SCM polling") ) {
    println "[Interrupting thread " + item.getName()+"]";
    item.interrupt();
  }
}

Like  this:
 
The script will locate all SCM polling threads and kill them.

5). Configure how often you wish to execute this job. Select "Build periodically" under the Build Triggers section.

Most of our jobs poll every 5 mins, so we run this every 14  - you must set it to what ever seems right in your context.



Comments