Images & Movies

   Posters

   Talks

   Publications

Ideal User Interfaces

Our Center has developed two novel user interface systems for microscopes that combine images and data sets from multiple inputs to form a coherent, complete view of biomedical experiments. Each interface gives scientists direct hands-on manipulation of the specimen, letting them pull on it in 3D while seeing and feeling how the system responds to pico-nanoscale forces. These systems put the scientist on the specimen, in control, while the experiment is happening:

• The same real-time 3D computer graphics cards used for cutting-edge games (augmented by stereo viewing) are employed to let the scientist see cells, cilia, and structures floating in space, providing an up-to-the-centisecond view of the action. Real-time 2D video is overlaid with 3D topography and motion, displayed overlapping in a consistent frame of reference.
• Robotic force-feedback displays from SensAble Technologies let the scientist push on DNA and cell membranes and feel the instantaneous response as chromatin unravels. Forces and motions are translated into the microscope’s frame of reference and displayed alongside the video and tracking.
• Semi-automatic manipulation modes complete the system, providing quantitative control over probe position and force profiles, letting scientists stretch fibrin fibers to the breaking point and then bring them back to the precise starting position to study how fully they recover.

These recent advances go beyond the control of a single microscope, providing combined tools that bring the advantages of multiple microscopes to bear in a consistent, integrated system to support experiments studying clotting disorders, cell division, gene expression, and the difference between cancer cells and healthy cells.

3DFM User Interface

Rendering of Volume, surface, and line data: Providing useful interfaces to the 3D force microscope requires effective display of the data obtained by the instrument. This includes the display of a volumetric scalar field (viscosity), a 1D curve in 3 space (the path of the driven magnetic bead), and surfaces with varying stiffness and certainty. These surfaces are obtained when the bead does not sample a region of space, either because it did not diffuse there (uncertain surface) or because it was pressed there but could not penetrate (surface with varying stiffness). There are no known effective techniques for the display of uncertain/varying surfaces. Computer Science graduate student Chris Weigle investigated the relative effectiveness of several potential methods of display (transparency, partially-transparent textures, hedgehog surfaces, animation, volume display, color, etc.) as his dissertation work. Each technique was evaluated against a set of questions (locate the maximum variance, identify the shape of the minimum surface, identify regions where variance exceeds a threshold, etc) to determine their relative strengths and weaknesses.

We implemented a prototype real-time volume rendering system in fall 1999 which uses hardware acceleration on our Reality Monster graphics engine in a way that enables the mixture of surfaces into the volume data. We have also implemented a prototype visualization tool on top of the Visualization ToolKit that displays a real-time trace of the bead trajectory (colored by estimated viscosity or other scalar data) combined with a display of the surface carved out by the bead's travel (indicating the boundary of the explored region). These results will be applied directly to the display of 3D force microscopy data, and we are working with a surgeon to apply them to the problem of displaying tumors that have been segmented from multi-modal MRI scans.

We are also investigating the presentation of multiple scalar and tensor fields in a volume as the magnetic bead samples different sections. Mapping Brownian motion and viscoelelastic properties of fluids and surfaces to the force levels and frequencies that are perceptible to humans. We are investigating both the perceptually-linear mappings of these senses and how the various perceptual channels interact.

The haptic presentation of the volume and surface data should be much more straightforward. If the control system produces stable, accurate measures of the force applied on the particle at sufficient rates, these can be displayed directly to the user. If this is not the case (e.g. if only position can be measured with sufficient update rate), we will use the point-to-point spring techniques described in [Mark1996] to couple the haptic probe to the microscope's probe, driving the force on the probe using a linear spring force.

Controlling ongoing experiments: During an experiment, the scientist will control the forces on (and/or position of) the bead using the Phantom force-feedback controller. This control can be fully manual, where either force-clamp or position-clamp control can be exerted by the user on the magnetic drive system. The control can also be semi-automatic, where the user specifies a location, path, area, or volume to be scanned and the computer completes the motion. In either case, the user's motion must be mapped to control commands and force feedback appropriate to the task must be presented to the user. We are developing haptic data presentation techniques and control methods to meet the challenges presented by this unique system.

Acquiring high-resolution 3D fluorescence data: The next prototype 3DFM system will include a 2D camera with which we will obtain 3D fluorescence images of the samples under study by sweeping the focus of the microscope through the volume, collecting slice-by-slice 2D data sets at known positions. Our microscope can servo to within 50nm of the desired locations, giving us a very good estimate of slice locations. To reduce the blurring impact of out-of-focus light, we will be using Computational Optical Sectioning Microscopy (COSM). This technique first characterizes the optical transfer function of a point-source emitter from within the lens system (by scanning a volume around the point source). It then convolves the corresponding kernel with the scanned volume to estimate the amount of out-of-plane light at each pixel; the procedure iterates to find the most likely solution within the volume.

Our 3DFM application will be collecting slice data at and near the plane of operation while the bead is moving through the volume. This provides us with additional partial information during the experiment. While recomputation of the entire COSM solution requires several minutes on a parallel computer, we anticipate being able to implement a partial-update solution that includes new information and displays the resulting volume data at interactive rates. We are starting with the XCOSM code developed at the NIH NCRR at Washington University in St. Louis.

Optimal Stochastic Estimation and Control: For our 3DFM to be useful as a general-purpose scientific instrument it must have a low-level system to control and observe the probe (magnetic bead). The simplest attack would involve classical stochastic control theory [ITT1975; Jacobs1993]. We have developed a low-level position feedback control system that is capable of 3-dimensional tracking of beads that are attached to cilia on living cells which are beating at up to 15Hz. However we believe the most interesting uses of the instrument described earlier will occasion correspondingly more interesting higher-level estimation and control challenges.

To support manipulation, force imaging, and estimation and visualization of viscoelastic and optical properties of the samples, the overall framework must estimate and control the hidden states of the probe (position {estimated based on photodiode signals} and its derivatives) while simultaneously estimating the unknown parameters associated with the medium and particles and the interactive user-specified forces.

Unfortunately neither the user nor the sample behavior can be completely characterized a priori. Further, the various processes will change continuously throughout operation. For example, users will intentionally change their mode of operation (sensing versus modifying) and their behavior both discretely and continuously. The control system must be adaptive, continuously estimating optimal process parameters and modes [Bar-Shalom1993; Jacobs1993].

One difficulty lies in the dynamics of the various processes. The dynamic models in each case include not only the device control and tracking components, but also the user and the (non-rigid) sample. For example as the probe approaches a cell wall, reactionary forces will influence the estimation of the viscoelastic properties of the medium. The change in dynamics could be interpreted as a change in the viscoelastic properties of the medium, or the presence of a nearby membrane. To address these issues we will incorporate both multi-modal stochastic estimation and control [Bar-Shalom1993], and concurrent localization and mapping (model building) approaches [Thrun1998; Feder1999; Fox1999].

Update:
As seen below, the 3DMF-UI is annotated with labels describing control of visualization and microscope parameters and connections to the various other parts of the 3DFM software suite. The labels and arrows are overlaid.
The 3D display (shown in a monoscopic view) displays the current location of a tracked bead as a green wire-frame sphere. A plane of video moves with the bead through the volume and shows objects in the specimen that lie on or near the focal plane passing through the bead. The sphere leaves a yellow trace as a trail, showing where the bead has moved during an experiment. This trace can be colored based on viscosity or another scalar field. A transparent shell is also drawn around the volume that has been “carved out” by the bead as it has moved along the trace; this shows the boundary of the region explored by the bead. Opaque strokes on this surface lie along directions of principal curvature.

nanoManipulator Interface

The nanoManipulator (nM) system provides a scientist with the ability to perform actions on objects as small as single molecules while at the same time quantitatively measuring both the surface shape and forces applied. The nM uses the ultra-sharp tip of a scanned-probe microscope (SPM) as tool both to scan and to modify samples. It uses advanced computer graphics to display the scanned surface to the user. A robot arm enables the user to feel and modify the surface.
Work in this period has focused on integrating the nanoManipulator with an optical fluorescence microscope, enabling the scientist to see what is happening during manipulation and providing overlaid display of both topography and force data. Additional modifications have been made to force-control modes to support manipulation without ever having scanned the surface. This combined system is being applied to fibrin studies and to the study of mucin molecules adsorbed onto surfaces.
The image to the right shows an example of an aligned set of images. The topography information (colored blue for lower areas and red for higher) is overlaid with an intensity fluorescence image. The inset shows the fluorescence image by itself.