Images & Movies

   Posters

   Talks

   Publications

Much of science is defining appropriate models to explain experimental results. Since each measurement contains some artifacts of the tool used to make it and each display of data biases the user towards a particular interpretation, scientists are constantly aware that preliminary interpretations must be validated against the actual data. The goal of this work is to combine techniques for acquisition, analysis, model building, hypothesis testing, and direct visual comparison into an experiment-driven system than enables the scientist to separate truth from illusion.

Image Analysis: Estimated models from images

The image above on the right shows a model of a tube that was extracted from an AFM image using MIDAG-produced CORE-tracking software. The model is drawn in green raised above the image from which it was extracted. This software is able to extract the 3D shape of the tube along with its width and curvature information at each point along the tube. This software enables us to extract and analyze tubular objects such as DNA, fibrin, and mucin from images. We have integrated the CORE-tracking code, originally developed for tracing blood vessels and other structures in medical images, into our microscopy applications.

The image on the left shows a trace of DNA on an AFM image as it passes through a protein. The boundary of the DNA as detected by the Resource code is shown as a thin blue line. The medial axis of the DNA is shown as a thin red line. The interpolated trace of the DNA and an estimated bend angle is indicated by a thin green line passing through the protein. The use of this code produced an order-of-magnitude decrease in analysis time and more accuracy than the manual estimation being routinely used before.

By feeding these extracted models to the AFM simulator (described next), a scientist can investigate which portions of the AFM scans are well-explained by the model. Using direct visual comparison between the model and scan (described below), a scientist can compare the model's fit and discover where it needs improving.

If you are interested in using our Tube Tracer model-from-image code, visit our software download page.

Visualization: Dual-Surface Display

This project aims to enable scientists to see the objects of two surfaces that share the same space and to reliably detect differences and similarities between the shapes of the surfaces. This comes about when comparing models to experiments. It has also found a use in tumor shape determination.
The image to the right shows the difference between two tumor segmentations, one done by a surgeon and one by an algorithm. The opaque two-tone inner surface is the intersection of the two tumor volumes, while the stroked outer surface is the union surface. Strokes are added to the transparent surface to make its shape perceptible, and shadows are cast onto the inner surface to make the inter-surface distance easier to estimate.

Imaging Simulator:
Estimated images from models

As described in our SPIE paper, hardware acceleration enables interactive calculation of imaging artifacts that would be expected from scanning a model with an atomic force microscope (AFM). This enables direct comparison between experimental results and expected microscope scans for both hand-made models and atom-coordinate data. The images to the right show this technique applied to a DNA/lac-repressor complex. The four images in the upper right corner show the result of imaging the crystallographics atom coordinates with increasingly coarse AFM tips. The bottom image shows a simplified model of a DNA strand wrapped around a protein; the image above it shows the AFM scan expected when this model is scanned. The image in the upper left shows an actual AFM scan of DNA wrapped around a protein. Such comparisons have enabled Dorothy Erie's chemistry group to determine which of several possible wrappings occur in actual experiments by showing that conformations which were not seen experimentally would have been resolvable with AFM had they occured, ruling out their being hidden by imaging or reconstruction artifacts.

We are working on making our AFM-image-of-model code available for outside use. If you are interested in using this code when it becomes available (hopefully Fall of 2002), email Russell Taylor to get on our distribution list.

Graphics Hardware Acceleration: Simulation and Optimization

Simulation: Here we have two goals: to reduce the mental load on the scientist during an atomic force microscopy (AFM) or fluorescence microscopy investigation and to reduce the impact of microscope imaging artifacts on image interpretation.
The first goal is met by providing an accurate display of what the AFM or fluorescence image would look like for a given geometric model of the specimen and assumed AFM tip shape and fluorescence lens system. This image can then be displayed alongside or combined with the actual experimental images as described in the dual-surface display subproject to enable direct visual comparison between model and experiment.
The second goal is met by enabling the scientists to use the AFM or fluorescence simulator to: (1) Develop intuition about what artifacts may be present in images by scanning different models with different tip shapes or optics. (2) Plan experiments based on what will be visible for a tip of a given radius. (3) Predict whether an expected feature should have been visible in a microscope scan by building a model that matches expectation and seeing whether the feature is visible in the simulated scan.
Optimization: The general problem of alignment involves comparing a predicted image of a model with an actual image. If the object model and pose are correct, we expect the predicted and actual images to be identical, or very close to it. The problem of finding the pose that gives the correct alignment is the challenge. To do this requires a measure of how well the simulated image is registered with the experimental image. Maximization of mutual information is a robust intensity-based approach that provides both the measure (mutual information) and the optimization technique.
We aim to accelerate this calculation to provide real-time model tracking during AFM of drift and model deformation in AFM/Fluorescence manipulation experiments. We also aim to use this to guide semi-automatic algorithms that adjust model parameters (tube radius and orientation, for example) that are then compared with experiment images after the AFM and/or fluorescence optical simulator have been run on them. This will enable gradient-descent optimization algorithms to improve model fit.

New Visualizations: Direct comparison of model and experiment

We are developing visualization techniques to enable the direct visual comparison of model and experiment. The image sequence above shows three techniques for the simultaneous display of an AFM scan of an adenovirus and an icosahedral model of the virus to determine the orientation of the virus that was scanned. The left image shows standard transperency, which has been shown to be not useful because it destroys the user's perception of the transparent surface. The center image uses a subsampled wire-frame view of the surface; it clearly shows both images but suffers from aliasing to the underlying scan mesh and from imprecise registration to the scan surface. The image to the right shows a partially-transparent texture applied to the surface; when combined with a 3D view of the surface and user-controlled viewpoint, this technique enables effective comparison between the two surfaces. We are continuing to investigate new techniques for such display and to validate them with user studies.

This work is also being applied to the visualization of uncertain surfaces (surfaces extracted from volume data where the gradient was low, multiple segmentations of a tumor surface by several radiologists, etc.).