The Problem

When debugging Image Processing code, it is often useful to look at images at intermediate processing stages. In the past this has involved inserting code to write out an image file, and then using Brains2 or another program to display the image.  Hans Johnson asked me to come up with a way to display images of intermediate results while running image processing software. In essence, come up with a way to make it possible to add debug statements that, instead of displaying text debugging messages, actually caused an image to be displayed. The requirements:

1. Minimal impact to the source code being tested.  Make any image display code a single line command.
2. Allow images to be replaced in a single display window. For example, run iterations of a registration algorithm, and display an image at each step, showing the progress of the registration.
3. Provide minimal interruption to the code being tested, e.g. don't display an image and then require the user to click a button to continue running.
4. Easy to use -- minimal setup required to instrument a program.
5. Display more than one image simultaneously.

The Solution: Client/Server

There are difficulties integrating a graphical display function into a program being debugged, especially if it is a command line/batch program, or a script.  If the program you're building doesn't already employ a GUI framework, there's a fundamental conflict in execution models:  Batch programs execute linearly, and GUI programs are event-driven.  The only choice if you wish to integrate a GUI function into a batch program is to pop up your GUI at some point in execution and run the GUI's event loop until the user presses a button, closing the GUI.

The solution I arrived at was to make the Image display GUI a separate program.  The program under test sends messages via Berkeley-style sockets telling the GUI program (DebugImageViewer) to display images.  Sockets can be used to communicate between programs running on a single computer, or to communicate with any computer attached to the Internet.  In practice, the local case is going to be the most common, because all of our computers, for security reasons, only open up TCP/IP ports for necessary network operation -- the NFS filesystems, Secure FTP, Secure sh, etc.

DebugImageViewer

DebugImageViewer is a program that utilizes ITK, KWWidgets, and VTK libraries to provide a GUI to display 3-Dimensional image volumes.  It's operation is very simple -- an Image is displayed, along with a slice to select the slice to display, and buttons to control which standard view is displayed.

The mouse can be used to pan, zoom, and control the brightness of the image. As described in the VTK Documentation the mouse behavior is as follows:

• Left Mouse button triggers window level events
• CTRL Left Mouse spins the camera around its view plane normal
• SHIFT Left Mouse pans the camera
• CTRL SHIFT Left Mouse dollys (a positional zoom) the camera
• Middle mouse button pans the camera
• Right mouse button dollys the camera.
• SHIFT Right Mouse triggers pick events 

DebugImageViewer Command Line Arguments

DebugImageViewer is the is the server side of the image display process.  Before starting the program you're testing, you'd run DebugImageViewer and park it on your computer display someplace convenient.  While running it listens for client requests to display an image. 

 --input  <filename>

Display the image named filename in one or more  image views.  This  allows for quick examination of an image file in any context.  Once the image is displayed, DebugImageViewer starts listening for client requests, and any image display requests will replace the initial image.  If no input image is displayed, the image views simply display a dummy all-black image.

--numviews <number>

Instead of displaying a single image [the default], DebugImageViewer can be told to display any number of images -- limited only by screen display space.

Client Side Usage

C++

To send image display requests from a C++ program you need to include DebugImageViewerClient.h in your program source code. Then, you need to create an instance of the client in your program:

DebugImageViewerClient disp;

The DebugImageViewer client doesn't actually open the connection to the DebugImageViewer program until it has been enabled:

disp.SetEnabled(true);

 Once this client object is enabled, to display an image, use the SendImage member function to send the image:

typedef itk::Image<char,3> ImageType;
ImageType::Pointer image;
// ... generate the image
disp.SendImage<itk::Image<char,3> >(image);

 If you've started DebugImageViewer server with more than one image window, you can specify which window to display your image in:

disp.SendImage(image1,0);
disp.SendImage(image2,1);

There's a method that will pause and wait for user input after every image is sent to the viewer. This is useful if you want to examine a particular image with the view controls.

disp.SetPromptUser(true);

 You can either create one instance of DebugImageViewerClient and use it repeatedly to send images.  Or, you can create an instance wherever it's most convenient, and let it get deleted when it goes out of scope.  Each instance of DebugImageViewerClient opens a socket connection to the server, which isn't particularly resource-intensive, but when you can, it's best to re-uses instances when you can.

Shell Scripts

Along with the DebugImageViewer program, a second program, DebugImageViewerClientTest is built and installed, and can be invoked from shell scripts to display images in a running instance of DebugImageViewer.

DebugImageViewerClientTest 0 DEBUG.nii.gz

This program always expects exactly 2 arguments -- a view number and a filename.

Current Implementation

The actual image data is written to the socket, and the server reads the image data from the socket.  The image dimension, spacing, orientation, and view number are written before the pixel data.  This information is used to create a corresponding itk::Image object to display, based on data coming through the pipe.

The implementation currently converts all images on the client side to a pixel type of unsigned char.  It also supports only 3D image data.

Limitations

DebugImageViewer is a very simple program, that doesn't take into account ever possible combination of events that might occur, the way, for example, a web server would.  It expects entire images to be sent through the pipe at once, from a single program.  You can 'break' it quite easily by running two programs at the same time that send images to it.  If you're lucky, the two programs won't both try and send images at the same time, but if not, the images will probably be scrambled, and the viewer may crash.

Likewise, the DebugImageViewerClient is not designed to be perfectly robust.  If you close the DebugImageViewer program, the next time your program attempts to send it an image it will exit, usually with the message 'Broken Pipe.'

Writing GUI Socket Servers with KWWidgets

It wasn't immediately apparent to me how to integrate listening on a socket for images to display into a KWWidgets application.  Logically the server works like this:

Do FOREVER
    wait for data on the socket
    process the data

This conflicts with the event-driven nature of KWWidgets GUI applications.  There is a solution, however: you can add the socket handle to the list of sockets or file handles that are checked in the main TCL event loop.  This comes out of KWWidgets hybrid nature:  KWWidgets is a high-level 'shell' API that works internally with Tcl, Tk and VTK to deliver a graphical user interface.

The KWWidgets way of handling event callbacks is a bit complicated.  When you write your application's GUI display classes using KWWidgets, your classes are 'wrapped' for TCL, meaning that C++ class methods are exposed to be called from TCL.  When you set up a KWWidget callback, you use the 'AddCommand' method, which binds the pertinent widget events with your handler by name, such that when an event occurs, the TCL string passed to AddCommand is executed by the TCL interpreter.  You can't have a callback in a KWWidgets application without it being wrapped for TCL.

This seems a little clumsy if you're used to XWindows or Windows MFC user interface APIs, but there is a valid reason for it:  At Kitware, who developed KWWidgets, their applications are written with scripting languages, usually TCL or Python.  This allows for a combination of rapid development of user interfaces, and high level computational process, but the low level number crunching can happen in C++.

This dual C++ / TCL nature of KWWidgets suggested a way to use TCL to set up a socket event handler:  Execute a little script to open the socket using the TCL socket command. This script defines a TCL function that will be called when the client writes to the socket.  This TCL function in turn calls one of my C++ member functions to actually load the image, using the TCL wrapping of that C++ member function.

void
vtkDebugImageViewer::
SetupServer()
{
  const char *viewerName = this->GetTclName();
  this->Script("socket -server SendFilename 19345");
  this->Script("proc SendFilename {sock addr port} {\n"
               "    fconfigure $sock -buffering line\n"
               "fileevent $sock readable [list SendToViewer $sock]\n"
               "}");
  this->Script("proc SendToViewer {sock} {\n"
               "if { [eof $sock] || [catch { gets $sock line} ]} {\n"
               " close $sock\n"
               "} else {\n"
               " %s SetImage $line\n"
               "}\n"
               "}",viewerName);
}

This script code is cribbed directly from this example of TCL server code from http://www.tcl.tk/ and plunked into my program.  The important bit is that my vtkDebugImageViewer class has been wrapped for TCL, and it has a member function defined thusly:

void vtkDebugImageViewer::SetImage(const char *s);

Since the TCL/Tk event loop is also the KWWidgets eventloop, the TCL 'fileevent' command sets up a callback to my wrapped C++ function whenever there's data to be read.