Rendering of Volume, surface, and line data: Providing useful interfaces to the 3D force microscope requires effective display of the data obtained by the instrument. This includes the display of a volumetric scalar field (viscosity), a 1D curve in 3 space (the path of the driven magnetic bead), and surfaces with varying stiffness and certainty. These surfaces are obtained when the bead does not sample a region of space, either because it did not diffuse there (uncertain surface) or because it was pressed there but could not penetrate (surface with varying stiffness). There are no known effective techniques for the display of uncertain/varying surfaces. Computer Science graduate student Chris Weigle is investigating the relative effectiveness of several potential methods of display (transparency, partially-transparent textures, hedgehog surfaces, animation, volume display, color, etc.) as his dissertation work. Each technique will be evaluated against a set of questions (locate the maximum variance, identify the shape of the minimum surface, identify regions where variance exceeds a threshold, etc) to determine their relative strengths and weaknesses.

We have implemented a prototype real-time volume rendering system in fall 1999. This prototype system uses hardware acceleration on our Reality Monster graphics engine in a way that enables the mixture of surfaces into the volume data. We have also implemented a prototype visualization tool on top of the Visualization ToolKit that displays a real-time trace of the bead trajectory (colored by estimated viscosity or other scalar data) combined with a display of the surface carved out by the bead's travel (indicating the boundary of the explored region). These results will be applied directly to the display of 3D force microscopy data, and we are working with a surgeon to apply them to the problem of displaying tumors that have been segmented from multi-modal MRI scans.

We are also investigating the presentation of multiple scalar and tensor fields in a volume as the magnetic bead samples different sections. Mapping Brownian motion and viscoelelastic properties of fluids and surfaces to the force levels and frequencies that are perceptible to humans. We are investigating both the perceptually-linear mappings of these senses and how the various perceptual channels interact.

The haptic presentation of the volume and surface data should be much more straightforward. If the control system produces stable, accurate measures of the force applied on the particle at sufficient rates, these can be displayed directly to the user. If this is not the case (e.g. if only position can be measured with sufficient update rate), we will use the point-to-point spring techniques described in [Mark1996] to couple the haptic probe to the microscope's probe, driving the force on the probe using a linear spring force.

Controlling ongoing experiments: During an experiment, the scientist will control the forces on (and/or position of) the bead using the Phantom force-feedback controller. This control can be fully manual, where either force-clamp or position-clamp control can be exerted by the user on the magnetic drive system. The control can also be semi-automatic, where the user specifies a location, path, area, or volume to be scanned and the computer completes the motion. In either case, the user's motion must be mapped to control commands and force feedback appropriate to the task must be presented to the user. We are developing haptic data presentation techniques and control methods to meet the challenges presented by this unique system.

Acquiring high-resolution 3D fluorescence data: The next prototype 3DFM system will include a 2D camera with which we will obtain 3D fluorescence images of the samples under study by sweeping the focus of the microscope through the volume, collecting slice-by-slice 2D data sets at known positions. Our microscope can servo to within 50nm of the desired locations, giving us a very good estimate of slice locations. To reduce the blurring impact of out-of-focus light, we will be using Computational Optical Sectioning Microscopy (COSM). This technique first characterizes the optical transfer function of a point-source emitter from within the lens system (by scanning a volume around the point source). It then convolves the corresponding kernel with the scanned volume to estimate the amount of out-of-plane light at each pixel; the procedure iterates to find the most likely solution within the volume.

Our 3DFM application will be collecting slice data at and near the plane of operation while the bead is moving through the volume. This provides us with additional partial information during the experiment. While recomputation of the entire COSM solution requires several minutes on a parallel computer, we anticipate being able to implement a partial-update solution that includes new information and displays the resulting volume data at interactive rates. We are starting with the XCOSM code developed at the NIH NCRR at Washington University in St. Louis.

Optimal Stochastic Estimation and Control: For our 3DFM to be useful as a general-purpose scientific instrument it must have a low-level system to control and observe the probe (magnetic bead). The simplest attack would involve classical stochastic control theory [ITT1975; Ja-cobs1993]. We have developed a low-level position feedback control system that is capable of 3-dimensional tracking of beads that are attached to cilia on living cells which are beating at up to 15Hz. However we believe the most interesting uses of the instrument described earlier will occasion correspondingly more interesting higher-level estimation and control challenges.

To support manipulation, force imaging, and estimation and visualization of viscoelastic and optical properties of the samples, the overall framework must estimate and control the hidden states of the probe (position {estimated based on photodiode signals} and its derivatives) while simultaneously estimating the unknown parameters associated with the medium and particles and the interactive user-specified forces.

Unfortunately neither the user nor the sample behavior can be completely characterized a priori. Further, the various processes will change continuously throughout operation. For example, users will intentionally change their mode of operation (sensing versus modifying) and their behavior both discretely and continuously. The control system must be adaptive, continuously estimating optimal process parameters and modes [Bar-Shalom1993; Jacobs1993].

One difficulty lies in the dynamics of the various processes. The dynamic models in each case include not only the device control and tracking components, but also the user and the (non-rigid) sample. For example as the probe approaches a cell wall, reactionary forces will influence the estimation of the viscoelastic properties of the medium. The change in dynamics could be interpreted as a change in the viscoelastic properties of the medium, or the presence of a nearby membrane. To address these issues we will incorporate both multi-modal stochastic estimation and control [Bar-Shalom1993], and concurrent localization and mapping (model building) approaches [Thrun1998; Feder1999; Fox1999].