Index

Deadlock?

Deadlocks can occur when two or more threads are locking resources in an order that results in all threads waiting indefinitely.

The simplest example is where Thread A locks Object A and then attempts to lock Object B, while another Thread B has Object B locked and is waiting to lock Object A. In this case, Thread A will never release Object A because it is waiting for Object B. This will never happen because Thread B will keep Object B locked indefinitely as it waits for Object A to become available.

wrapper.check.deadlock
Compatibility :3.5.0
Editions :Professional EditionStandard EditionCommunity Edition (Not Supported)
Platforms :WindowsMac OSXLinuxIBM AIXFreeBSDHP-UXSolarisIBM z/Linux

NOTE

Thread deadlock checks require that at least Java version 1.5 is used. Older JVMs will ignore the checks. It is possible to detect deadlocks involving "monitor" based thread locks (using synchronized) starting with Java 1.5. Deadlocks involving "owned" or "reentrant" thread locks (using ReentrantLock) can also be detected starting with Java 1.6.

With Java 9, if a custom runtime image doesn't include the java.management module, deadlock checks will be disabled.

This wrapper.check.deadlock property, in combination with the other properties,

are used to configure how the Wrapper monitors a JVM for deadlocked threads. This can be very useful to detect and then work around potentially fatal problems that would otherwise be difficult if not impossible to work around.

Checking for Deadlocks is fairly quick, but it does involve briefly locking and taking a snapshot of all threads, so this property is set to FALSE by default.

Example: (Deadlock Check: Off) 
wrapper.check.deadlock=FALSE

Simple Example:

Please read over the property details below, but the following simple example will configure the Wrapper to log the location of a deadlock and then immediately restart the JVM.

Example: 
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=60
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL

wrapper.check.deadlock.interval
Compatibility :3.5.0
Editions :Professional EditionStandard EditionCommunity Edition (Not Supported)
Platforms :WindowsMac OSXLinuxIBM AIXFreeBSDHP-UXSolarisIBM z/Linux

The wrapper.check.deadlock.interval property makes it possible to control the interval at which the Wrapper looks for deadlocks in an application. It can be set to any interval starting with once per second, but the default value is "60" seconds (once per minute). In general, for applications that are known to be stable, it may be fine to greatly decrease the frequency of these deadlock checks.

Example: (once every 60 seconds) 
wrapper.check.deadlock.interval=60

wrapper.check.deadlock.action
Compatibility :3.5.0
Editions :Professional EditionStandard EditionCommunity Edition (Not Supported)
Platforms :WindowsMac OSXLinuxIBM AIXFreeBSDHP-UXSolarisIBM z/Linux

The wrapper.check.deadlock.action property makes it possible to control what the Wrapper does when a deadlock is detected. The default action is to RESTART.

Example: 
wrapper.check.deadlock.action=RESTART

Possible actions are:

  • DEBUG:

    will cause a debug message to be logged. This is only really useful in helping to understand when the action is fired.

  • STATS (Since ver. 3.5.52, Standard /Professional Edition):

    will print performance statistics.

  • DUMP:

    will invoke a thread dump.

  • GC (Since ver. 3.5.7):

    will invoke a full garbage collection sweep in the JVM. Be aware that doing this frequently can affect performance of the JVM, as a full sweep will often cause all threads to freeze for the duration of the GC.

  • RESTART:

    will stop the current JVM and then restart a new invocation.

  • SHUTDOWN:

    will stop the JVM as well as the Wrapper.

  • USER_<n> (Professional Edition):

    will cause a user-defined event to be fired. This can be either the sending of an email, or the execution of an external system command. The command could be anything from performing cleanup operations to raising an SNMP trap.

  • PAUSE:

    will pause the Java application if pausing is enabled and the JVM is running. See the wrapper.pausable property for details.

  • RESUME:

    will resume the Java application if it is in a paused state. This could be used if the JVM is not stopped when paused. See the wrapper.pausable property for details.

  • SUSPEND_TIMEOUTS_<n> (Since ver. 3.5.40, Standard /Professional Edition):

    Tells the Wrapper to suspend all timeouts used when the JVM is not responding. <n> specifies the number of seconds to suspend timeouts, and should be in the range 1-3600 (1h). This could be used if the Java application needs to perform a long blocking task and avoid having the Wrapper considering the application as unresponsive.

    Timeouts can also be suspended via other action properties, the command file or the Java WrapperManager.suspendTimeouts() method.

    If several requests to suspend timeouts are made, the number of seconds specified by each request will not be summed. Instead, the newly specified time will replace the remaining suspension time if it is longer, and will be ignored otherwise.

  • RESUME_TIMEOUTS (Since ver. 3.5.40, Standard /Professional Edition):

    Tells the Wrapper to resume all timeouts previously suspended.

    Timeouts can also be resumed via other action properties, the command file or the Java WrapperManager.resumeTimeouts() method.

  • SUCCESS (Since ver. 3.5.5):

    will tell the Wrapper to reset its internal failed invocation count and count the current JVM invocation as "successful". This is probably not useful in this context, but is here for consistency with other properties.

  • NONE:

    will tell the Wrapper to log the fact that a deadlock was detected, but it will not actually do anything.

NOTE

Note that actions that do not restart the JVM, like "NONE", will repeatedly be fired each time the deadlock check is made.

Chaining Multiple Actions:

It is possible to specify more than one action by separating them with a space or comma. When more than one action is specified, they will be executed in rapid succession in the order specified.

The following example will perform a thread dump and then restart the JVM.

Example: 
wrapper.check.deadlock.action=DUMP,RESTART

wrapper.check.deadlock.output
Compatibility :3.5.0
Editions :Professional EditionStandard EditionCommunity Edition (Not Supported)
Platforms :WindowsMac OSXLinuxIBM AIXFreeBSDHP-UXSolarisIBM z/Linux

The wrapper.check.deadlock.output property makes it possible to control what information the Wrapper causes to be logged when a deadlock is detected. The default output level is FULL.

Example: 
wrapper.check.deadlock.output=FULL

Possible output levels are:

  • FULL:

    will cause the WrapperManager class within the JVM to output a report that includes full stack traces for the threads involved in the deadlock.

    Output Example of "FULL": 
    INFO   | jvm 1    | WrapperManager Error: Found 2 deadlocked threads!
INFO   | jvm 1    | WrapperManager Error: =============================
INFO   | jvm 1    | WrapperManager Error: "Locker-2" tid=18
INFO   | jvm 1    | WrapperManager Error:   java.lang.Thread.State: BLOCKED
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockSecond(DeadLock.java:64)
INFO   | jvm 1    | WrapperManager Error:       - waiting on <0x000000002fcac6db> (a java.lang.Object) owned by "Locker-1" tid=17
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockFirst(DeadLock.java:83)
INFO   | jvm 1    | WrapperManager Error:       - locked <0x0000000029c56c60> (a java.lang.Object)
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.access$100(DeadLock.java:22)
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock$1.run(DeadLock.java:42)
INFO   | jvm 1    | WrapperManager Error:
INFO   | jvm 1    | WrapperManager Error: "Locker-1" tid=17
INFO   | jvm 1    | WrapperManager Error:   java.lang.Thread.State: BLOCKED
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockSecond(DeadLock.java:64)
INFO   | jvm 1    | WrapperManager Error:       - waiting on <0x0000000029c56c60> (a java.lang.Object) owned by "Locker-2" tid=18
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.lockFirst(DeadLock.java:83)
INFO   | jvm 1    | WrapperManager Error:       - locked <0x000000002fcac6db> (a java.lang.Object)
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock.access$100(DeadLock.java:22)
INFO   | jvm 1    | WrapperManager Error:     at org.tanukisoftware.wrapper.test.DeadLock$1.run(DeadLock.java:42)
INFO   | jvm 1    | WrapperManager Error:
INFO   | jvm 1    | WrapperManager Error: =============================
STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.
  • SIMPLE:

    will cause the WrapperManager class within the JVM to output a report which includes only a brief summary naming the threads and objects involved in the deadlock. In many cases, this is enough, especially for known problems.

    Output Example of "SIMPLE": 
    INFO   | jvm 1    | WrapperManager Error: Found 2 deadlocked threads!
INFO   | jvm 1    | WrapperManager Error: =============================
INFO   | jvm 1    | WrapperManager Error: "Locker-2" BLOCKED waiting on a java.lang.Object owned by "Locker-1"
INFO   | jvm 1    | WrapperManager Error: "Locker-1" BLOCKED waiting on a java.lang.Object owned by "Locker-2"
INFO   | jvm 1    | WrapperManager Error: =============================
STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.
  • NONE (Since ver. 3.5.16):

    will cause the WrapperManager class within the JVM to suppress all output. This may be desired for production systems when a problem is known but you don't want or need to maintain too much information in the logs. The Wrapper process, as in the other options, will always log a single entry to provide a reason why the triggered action takes place.

    Output Example of "NONE": 
    STATUS | wrapper  | A Thread Deadlock was detected in the JVM.  Restarting JVM.