Running the TestWrapper Example Application

This page will describe the TestWrapper Example Application, which is shipped with the Wrapper to demonstrate several of its failure recovery features.

Running in a Console

To run the application in a console, execute the following script located in the bin directory:

Windows: 
TestWrapper.bat
Unix: 
./testwrapper console

NOTE

Prior to version 3.0.5, the testwrapper script on UNIX tried to write a PID file to the /var/run directory. Non-root users would need to change the PIDDIR variable of the script to a directory having appropriate permissions. This is no longer an issue since version 3.0.5 as the Wrapper is writting the PID file in the same location as the script.

When the application starts, the following window will appear:

You should also see the following output in the console:

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()

Let's start with the console output. The first column of data displays the source of the output. Lines starting with "wrapper" are coming from the native Wrapper application. Lines starting with "jvm 1" are coming from the output of the JVM. The number after "jvm" indicates the number of JVM(s) launched since the Wrapper was started. We will see how that works when we start playing with the TestWrapper's interface.

By default all console output is also logged to logs/wrapper.log. However, the format of log file is configured to also include the log level and the timestamps for each message being logged.

Output in the log file: 
STATUS | wrapper  | 2001/12/06 16:18:27 | --> Wrapper Started as Console
STATUS | wrapper  | 2001/12/06 16:18:28 | Launching a JVM...
INFO   | jvm 1    | 2001/12/06 16:18:29 | Initializing...
INFO   | jvm 1    | 2001/12/06 16:18:30 | Wrapper (Version 3.x.x)
INFO   | jvm 1    | 2001/12/06 16:18:30 | 
INFO   | jvm 1    | 2001/12/06 16:18:30 | start()

It is possible to reconfigure separately the format as well as the log level for each output. This can be done using the following properties:

By default, the TestWrapper Application presents you with an interface showing several buttons to test various functionalities of the Wrapper. Each button is described below with the output you will see in the console if the button is activated.

As an alternative to the GUI, the TestWrapper Application can be controlled exclusively from a terminal. This mode, called 'console', has the same functionalities as the GUI, but the application will wait for user inputs, and the tests can be triggered by writting commands in the terminal instead of by clicking on buttons.

TestWrapper Application run in 'console' mode: 
jvm 1    | WrapperManager: Initializing...
jvm 1    | TestWrapper: Initializing...
jvm 1    | TestWrapper: start()
jvm 1    | TestWrapper: ActionServer Enabled.
jvm 1    | TestWrapper:   Telnet localhost 9999
jvm 1    | TestWrapper:   Commands:
jvm 1    | TestWrapper:     S: Shutdown
jvm 1    | TestWrapper:     H: Expected Halt
jvm 1    | TestWrapper:     R: Restart
jvm 1    | TestWrapper:     D: Thread Dump
jvm 1    | TestWrapper:     U: Unexpected Halt (Simulate crash)
jvm 1    | TestWrapper:     V: Access Violation (Actual crash)
jvm 1    | TestWrapper:     G: Make the JVM appear to be hung.
jvm 1    | TestWrapper:
jvm 1    |
jvm 1    | Start prompting for actions.
jvm 1    | Input an action ('help' for a list of actions):
help
jvm 1    | Read action: help
jvm 1    |
jvm 1    | [ACTIONS]
jvm 1    |    help                     : Shows this help message
jvm 1    |   Actions which should cause the Wrapper to exit cleanly:
jvm 1    |    stop0                    : Calls WrapperManager.stop(0)
jvm 1    |    exit0                    : Calls System.exit(0)
jvm 1    |    stopimmediate0           : Calls WrapperManager.stopImmediate(0)
jvm 1    |    stopandreturn0           : Calls WrapperManager.stopAndReturn(0)
jvm 1    |   Actions which should cause the Wrapper to exit in an error state:
jvm 1    |    stop1                    : Calls WrapperManager.stop(1)
jvm 1    |    exit1                    : Calls System.exit(1)
jvm 1    |    nestedexit1              : Calls System.exit(1) within WrapperListener.stop(1) callback
jvm 1    |    stopimmediate1           : Calls WrapperManager.stopImmediate(1)
jvm 1    |   Actions which should cause the Wrapper to restart the JVM:
jvm 1    |    access_violation_native  : Calls WrapperManager.accessViolationNative()
jvm 1    |    appear_hung              : Calls WrapperManager.appearHung()
jvm 1    |    halt0                    : Calls Runtime.getRuntime().halt(0)
jvm 1    |    halt1                    : Calls Runtime.getRuntime().halt(1)
jvm 1    |    restart                  : Calls WrapperManager.restart()
jvm 1    |    restartandreturn         : Calls WrapperManager.restartAndReturn()
jvm 1    |   Additional Tests:
jvm 1    |    appear_slow_1            : Calls WrapperManager.appearSlow(1)
jvm 1    |    appear_slow_5            : Calls WrapperManager.appearSlow(5)
jvm 1    |    appear_slow_reset        : Calls WrapperManager.appearSlow(0) to return to normal
jvm 1    |    deadlock                 : Executes some deadlocking code to test deadlock detection
jvm 1    |    outofmemory              : Simulates an OutOfMemoryError being thrown
jvm 1    |    ignore_events            : Makes this application ignore control events.
jvm 1    |    dump                     : Calls WrapperManager.requestThreadDump()
jvm 1    |    deadlock_out             : Deadlocks the JVM's System.out and err streams.
jvm 1    |    users                    : Start polling the current and interactive users.
jvm 1    |    groups                   : Start polling the current and interactive users with groups.
jvm 1    |    console                  : Prompt for actions in the console.
jvm 1    |    idle                     : Do nothing just run in idle mode.
jvm 1    |    properties               : Dump all System Properties to the console.
jvm 1    |    configuration            : Dump all Wrapper Configuration Properties to the console.
jvm 1    |    gc                       : Perform a GC sweep.
jvm 1    |    is_professional          : Displays whether or not this is a Professional Edition Wrapper.
jvm 1    |    is_standard              : Displays whether or not this is at least a Standard Edition Wrapper.
jvm 1    |    exec <cmd>               : Executes a managed child process.
jvm 1    |    exec_detached <cmd>      : Executes a detached child process.

To launch the application in 'console' mode, you should give it an additional 'console' argument. On UNIX, this argument should not be confused with the 'console' command passed to the script (the word 'console' should thus be repeated twice when launching the script).

Windows: 
TestWrapper.bat console
Unix: 
./testwrapper console console

You can stop the application anytime by closing the window or by using the "Stop()" and "Exit()" buttons. You may also press CTRL-C in the terminal to request a clean shutdown.

Testing Wrapper Features (Internal Exits and Problems)

This section will demonstrate how the Wrapper responds to various events. Each of these events can be tested by pressing the buttons of the user interface.

Stop

Exits the JVM cleanly by calling WrapperManager.stop(0).

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
jvm 1    | stop(0)
wrapper  | <-- Wrapper Stopped

As you can see, the application had its stop() method called and was stopped cleanly.

Exit

This Exit button tests the Wrapper's ability to trap an unexpected call to System.exit() and stop the application gracefully.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
jvm 1    | stop(0)
jvm 1    | here
wrapper  | <-- Wrapper Stopped

As an exercise, edit your conf/wrapper.conf file and uncomment the property wrapper.disable_shutdown_hook=TRUE. (Make sure to comment it back out when you are done.)

Now rerun the TestWrapper Application.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | JVM exited unexpectedly.
wrapper  | Launching a JVM...
jvm 2    | Initializing...
jvm 2    | Wrapper (Version 3.x.x)
jvm 2    |
jvm 2    | start()

You will notice that the Wrapper no longer catches the call to System.exit(), so it thinks that the JVM crashed and restarts it.

Halt

This Halt button calls a very low-level method to stop the JVM forcibly. In this case, shutdown hooks are not executed and the Wrapper has no way to catch them.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | JVM exited unexpectedly.
wrapper  | Launching a JVM...
jvm 2    | Initializing...
jvm 2    | Wrapper (Version 3.x.x)
jvm 2    |
jvm 2    | start()

Notice that the Wrapper thinks that the JVM crashed and restarts it.

Request Restart

Restarts the JVM cleanly by calling WrapperManager.restart()

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | JVM requested a restart.
jvm 1    | stop(0)
wrapper  | Launching a JVM...
jvm 2    | Initializing...
jvm 2    | Wrapper (Version 3.x.x)
jvm 2    |
jvm 2    | start()

As you can see, the application had its stop() method called and was restarted cleanly.

Native Access Violation

The Java Service Wrapper includes a lightweight native library that is used to handle system events. The library also contains a native method, which is used for testing, to cause an access violation in native code due to a "NULL" reference.

Depending on the JVM, this will give different output.

Console Output (Oracle JVM): 
wrapper  | -->jvm 1    | WrapperManager: WARNING: Attempting to cause an access violation...
jvm 1    | #
jvm 1    | # A fatal error has been detected by the Java Runtime Environment:
jvm 1    | #
jvm 1    | #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007fffc9aee46f, pid=14352, tid=12948
jvm 1    | #
jvm 1    | # JRE version: Java(TM) SE Runtime Environment (8.0_45-b15) (build 1.8.0_45-b15)
jvm 1    | # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode windows-amd64 compressed oops)
jvm 1    | # Problematic frame:
jvm 1    | # C  [wrapper.dll+0xe46f]  Java_org_tanukisoftware_wrapper_WrapperManager_accessViolationInner+0x1f
jvm 1    | #
jvm 1    | # Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
jvm 1    | #
jvm 1    | # An error report file with more information is saved as:
jvm 1    | # C:\wrapper\bin\hs_err_pid14352.log
jvm 1    | #
jvm 1    | # If you would like to submit a bug report, please visit:
jvm 1    | #   http://bugreport.java.com/bugreport/crash.jsp
jvm 1    | # The crash happened outside the Java Virtual Machine in native code.
jvm 1    | # See problematic frame for where to report the bug.
jvm 1    | #
wrapper  | JVM exited unexpectedly.
wrapper  | Launching a JVM...
jvm 2    | WrapperManager: Initializing...
jvm 2    | TestWrapper: Initializing...

As you can see, an access violation caused the JVM to display an error dump and then terminate. The Wrapper immediately detects this and after a 5 seconds pause, restarts the application in a new JVM, as can be seen with the jvm 2. All available information about the crash has been logged for future reference.

Simulate JVM Hang

This test causes the Wrapper to think that the JVM has become hung. After 30 seconds, the Wrapper times out and decides that the JVM will not come back and restarts it.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
jvm 1    | WARNING: Making JVM appear to be hung...
wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
wrapper  | JVM did not exit on request, terminated
wrapper  | Launching a JVM...
jvm 2    | Initializing...
jvm 2    | Wrapper (Version 3.x.x)
jvm 2    |
jvm 2    | start()

Testing Wrapper Features (External Exits)

The above section covered all of the ways that an application can exit or die of its own will. There are other external factors that the Wrapper can also protect itself against.

CTRL-C

Most Java applications die rather abruptly if the user presses CTRL-C, logs out of Windows, etc. You can work around some of these issues with a Shutdown Hook. The Wrapper implements this by using a native library to directly capture the system signals. This makes it possible to have a Java application installed as a Windows Service without it being stopped when a user logs out.

As a test, try pressing CTRL-C in the console (command window).

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | CTRL-C trapped.  Shutting down.
jvm 1    | stop(0)
wrapper  | <-- Wrapper Stopped

You can also see variations on this by logging out when the console application is still running. The Wrapper will correctly terminate the application.

Console Output: 
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    | 
jvm 1    | start()
wrapper  | User logged out.  Shutting down.
jvm 1    | stop(0)
wrapper  | <-- Wrapper Stopped

On Unix platforms, you can only press CTRL-C if the script used to launch the Wrapper was passed the 'console' command.

If the Wrapper was launched using the 'start' command, then it has been spawned in a different process as a Service/Daemon, so you can't access the console. To stop the spawned instances of the Wrapper, run the same script used to launch the Wrapper, only this time specify the 'stop' command:

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | Shutting down.
jvm 1    | stop(0)
wrapper  | <-- Wrapper Stopped

Process Termination

It is also possible, though unlikely, that the application process could be killed by another application. This test also simulates cases where the JVM dies very suddenly of its own accord. On UNIX systems, you can kill the Java process with a kill -9 signal. On Windows systems, open up the Task Manager, and then select the Process tab. After that, find the java.exe process to kill and then kill it.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
wrapper  | JVM exited unexpectedly.
wrapper  | Launching a JVM...
jvm 2    | Initializing...
jvm 2    | Wrapper (Version 3.x.x)
jvm 2    |
jvm 2    | start()

The Wrapper will restart the JVM without any problems.

If rather than killing the Java process, the Wrapper process is killed, then WrapperManager will be in recovery mode. The WrapperManager inside the Java process reacts to the loss of the Wrapper by assuming that the Java process is abandoned. This leads to the JVM being shutdown cleanly 30 seconds after the Wrapper process died.

System Under High Load or System Suspend

There are cases where either the Wrapper, the JVM or both will be denied CPU cycles for an extended period of time. This can happen if a resource unfriendly process starts taking 100% of the CPU for a period of time. It is also possible if the user suspends the system to RAM or disk. When the system is resumed, all running applications will have experienced some missing time. The Wrapper is capable of recognizing that this has happened making it possible to avoid unneeded restarts of the JVM process. The following output shows what happens when the System is suspended to RAM for 255 seconds before being resumed.

Console Output: 
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Initializing...
jvm 1    | Wrapper (Version 3.x.x)
jvm 1    |
jvm 1    | start()
jvm 1    | JVM Process has not received any CPU time for 255 seconds.  Extending timeouts.
wrapper  | Wrapper Process has not received any CPU time for 255 seconds.  Extending timeouts.

As of Wrapper version 3.1.0, a new timer mechanism was added which allows the Wrapper to handle cases where it is running in a CPU starved state reliably because the tick count is incremented at a rate that reflects the amount of CPU being received, rather than being absolute. This means that timeouts due to high loads are very unlikely. Just make sure that the wrapper.use_system_time property is set to FALSE for the timer mechanism to be enabled.

Running as a Windows Service

When running as a Windows Service, the TestWrapper Application will be unable to show its user interface. This prevents you from launching the tests described above. We can, however, do a few tests to verify that the Wrapper acts as a Windows Service correctly.

The first thing you will need to do is get the TestWrapper Application installed as a Windows Service by running the following command:

Command: 
InstallTestWrapper-NT.bat
You will see the output: 
wrapper  | TestWrapper Application installed.

Once the TestWrapper Application is installed as a service, we will want to start it.

Command: 
net start TestWrapper
You will see the output: 
The TestWrapper Application service is starting.
The TestWrapper Application service was started successfully.

The service can be uninstalled by executing the following command: 

UninstallTestWrapper-NT.bat
If the service is running, you will see the output: 
wrapper  | Service is running.  Stopping it...
wrapper  | Waiting to stop...
wrapper  | TestWrapper Application stopped.
wrapper  | TestWrapper Application removed.

If you look at the content of logs/wrapper.log, you will see an output very similar to that of the console (except that, this time, the start message informs you that the application is being started as a service instead of as a console).

wrapper.log 
wrapper  | 2001/12/06 17:34:21 | --> Wrapper Started as Service
wrapper  | 2001/12/06 17:34:21 | Launching a JVM...
jvm 1    | 2001/12/06 17:34:22 | Initializing...
jvm 1    | 2001/12/06 17:34:22 | Wrapper (Version 3.x.x)
jvm 1    | 2001/12/06 17:34:22 |
jvm 1    | 2001/12/06 17:34:22 | start()

Here are the results of several more actions that can take place on a Windows Service:

User Logging out and then back in

A log entry is made when the user logs out, but the service is unaffected.

wrapper.log 
wrapper  | 2001/12/06 17:39:39 | --> Wrapper Started as Service
wrapper  | 2001/12/06 17:39:40 | Launching a JVM...
jvm 1    | 2001/12/06 17:39:40 | Initializing...
jvm 1    | 2001/12/06 17:39:40 | Wrapper (Version 3.x.x)
jvm 1    | 2001/12/06 17:39:40 |
jvm 1    | 2001/12/06 17:39:41 | start()
wrapper  | 2001/12/06 17:40:07 | User logged out.  Ignored.
jvm 1    | 2001/12/06 17:40:07 | controlEvent(202)

Restarting the machine

This will result in a "logout" signal followed by a "shutdown" signal. The service will be shutdown gracefully and then come back after the machine restarts.

A log entry is made when the user logs out, but the service is unaffected.

wrapper.log 
wrapper  | 2001/12/06 17:41:04 | --> Wrapper Started as Service
wrapper  | 2001/12/06 17:41:05 | Launching a JVM...
jvm 1    | 2001/12/06 17:41:05 | Initializing...
jvm 1    | 2001/12/06 17:41:05 | Wrapper (Version 3.x.x)
jvm 1    | 2001/12/06 17:41:05 |
jvm 1    | 2001/12/06 17:41:05 | start()
wrapper  | 2001/12/06 17:41:25 | User logged out.  Ignored.
jvm 1    | 2001/12/06 17:41:26 | controlEvent(202)
wrapper  | 2001/12/06 17:41:27 | Machine is shutting down.
jvm 1    | 2001/12/06 17:41:27 | controlEvent(203)
jvm 1    | 2001/12/06 17:41:28 | stop(0)
wrapper  | 2001/12/06 17:41:29 | <-- Wrapper Stopped
wrapper  | 2001/12/06 17:44:12 | --> Wrapper Started as Service
wrapper  | 2001/12/06 17:44:12 | Launching a JVM...
jvm 1    | 2001/12/06 17:44:17 | Initializing...
jvm 1    | 2001/12/06 17:44:21 | Wrapper (Version 3.x.x)
jvm 1    | 2001/12/06 17:44:21 |
jvm 1    | 2001/12/06 17:44:23 | start()

Please read this page for detailed information on how to control the Wrapper as a Windows Service.

Running as a Unix Daemon

The Wrapper can be launched using the 'start' command to run in the background. If it is launched from a terminal that is unable to display the GUI, the application will switch to its console mode, but it won't be so useful because the Daemon can't be controlled with the console.

Unix: 
./testwrapper start

Once the application is started, the UI of the TestWrapper Application being run as a daemon can be used exactly like when running as a console application.

To stop instances of the Wrapper running as a daemon, use the same script with the 'stop' command.

Unix: 
./testwrapper stop

Please read this page for a complete description of the commands that can be used to control the Wrapper being run as Unix Daemon.