ImageSurfer

The ImageSurfer Roadmap is a big-picture view of functionality we expect ImageSurfer to include in short-term and long-term future.
We provide this Roadmap to show where ImageSurfer is heading, and to give you a chance to shape that direction. Make suggestions, and even contribute code to ImageSurfer. Whatever way you choose to provide feedback, we're looking forward to hearing it!

The Roadmap has three main components:

1. Strengthen ImageSurfer by developing a new core.
2. Make ImageSurfer more easily usable.
3. Provide new tools for image restoration and analysis.

We want to rewrite ImageSurfer's core, to eliminate memory limitations, and to provide a plug-in structure allowing both developers and end users to implement their own solutions. We will use Qt in combination with OverView (a stripped-down version of ParaView), to combine good user interface design with the plug-in architecture already developed for OverView. OverView's architecture will also enable ImageSurfer to efficiently run on a wide range of machines, from laptops to supercomputers.

Diagram of ImageSurfer 2.0 architecture. ImageSurfer 2.0 will be composed of custom plug-ins and application code running on top of the existing OverView/ParaView/VTK architecture. External developers will be able to write plug-ins using the same interface (or a Python-based interface) to augment ImageSurfer with new algorithms and techniques. Other C++-based libraries such as μManager can be linked through plug-ins.

New graphical user interface, using whenever practical a question/answer interface to guide users. Several task-oriented wizard will also be available to guide beginners to achieve a specific goal. 
For example, the "deconvolution wizard" will guide the user through the process of selecting an appropriate deconvolution methodology for his/her specific dataset.

Contextual help. Tooltips appearing on the screen when the user moves his mouse over an interface element will provide the user with information about the element he is currently interested in. In addition, we will provide a hint (by displaying a special icon in the tooltip) that more documentation is available for that element. Thus, if the user finds that the information provided in the tooltip is not sufficient, he/she is pointed to the next level of help.

Support for more microscope-specific formats. Due to the large number of microscope-specific file formats, loading datasets into ImageSurfer can be difficult for novice users. ImageSurfer already supports several microscope-specific formats; we will expand this further.

Support 3D-specific devices. Interacting with complex 3D representations is inhibited by the two dimensional nature of interaction and display hardware. 3D images are usually viewed on flat screens; the sense of 3D is conveyed by a variety of features, including occlusion, perspective, and shading based on directional illumination. However, this does not produce the same impression achieved when we look around a room; that sense of depth also requires binocular disparity. The required software support for stereo viewing is built into the VTK toolkit that ImageSurfer uses for display. We will support a variety of stereo techniques (anaglyphic red/cyan, frame-sequential, or line-interleaved) using consumer-grade hardware.

User-friendly updater. Users will be able retrieve and install updates directly through an updater within ImageSurfer. The updater will also obtain and update plug-in software. It will display available plug-ins along with a description of their function; with a single click, users will be able to get more information, install or update a plug-in.

We want to extend ImageSurfer's deconvolution tools. Clarity, our open-source implementation of image deconvolution algorithms, will be fully integrated into ImageSurfer and extended (more on Clarity HERE)

We will make available:

1. Point-spread-function (PSF) specification/estimation using four distinct methods:

a. Analytic specification of the PSF: Analytic PSF models will be provided, based on known microscope parameters (i.e. numerical aperture, media index of refraction, wavelength, voxel size).
b. Empirical estimation of the PSF: Estimation of the PSF through measurements of microscopic beads. Averaging capabilities for multiple measurements (to increase signal to noise ratio) will be provided.
c. Parametric-Empirical estimation of the PSF: Analytic specification of the PSF relies on an accurate parameterization of the PSF, empirical estimation on accurate measurements. To combine the benefits of both methods, we will implement a maximum-likelihood estimator to identify the parameters of the analytical model based on the empirical measurements of the PSF, effectively combining approaches #a and #b above.
d. PSF Estimation through blind deconvolution: We will implement blind deconvolution methods to provide an alternative data-driven method to estimate a PSF without the need for explicit measurements of the microscope's impulse response. In addition, we will provide non-parametric estimation of the PSF (based on bandwidth constraints and symmetry assumptions) as well as parametric estimation, analogous to (c)).

2. Non-blind deconvolution algorithms, which use the specified or estimated PSF from (1).

3. Blind deconvolution algorithms, which combine PSF estimation and image deconvolution.

We want to provide an iterative method combining the user's ability to recognize objects within complex images, with powerful automatic segmentation algorithms. These methods allow the user to first isolate a volume of interest (VOI) using both manual and semi-automatic tools. This userdefined volume will approximate the object of interest (depending on the algorithm used by confining the search-space to a sub-region of the overall image volume, by providing an initial rough segmentation, or both). Subsequent automatic segmentation will produce a precise segmentation of the object of interest.

Flow chart of image processing and analysis using ImageSurfer 2.0. After preprocessing of an image using deconvolution and image filtering, objects of interest are defined on the image. Segmentation algorithms are based on the volume of interest definition. Upon inspecting the output of the segmentation algorithm and its parameter sensitivity, segmentation settings are refined, in an iterative manner: subsequently new volumes of interest can be defined based on the segmentation results, allowing for refined object extraction and  ultimately, quantification of image information.

The figure below shows how our two-stage approach performs at segmenting a presynaptic vesicle from electron tomographic material. The user only has to specify a few "seed" points within the vesicle on one single slice. The algorithm then automatically finds the 3D vesicle boundaries, defining the VOI. Tests confirm that the result is robust to seed placement, as long as the seeds are placed strictly inside the vesicle. Segmentation is then obtained automatically through three-dimensional seeded region-growing, where the seed areas were obtained by watershed segmentation performed on the two-dimensional manifold described by the vesicle surface, instead of directly in voxel space.

Semi-automatic segmentation of a presynaptic vesicle. A, raw image. B, The user places seed points in the interior of the vesicle. C, D The vesicle is automatically detected in 3D space, defined by its membrane. E, 3D watershed segmentation is performed, defining individual proteins or proteins cluster. F, G 3D views of the segmented objects.

Evaluating the quality of a segmentation is essential. We want to provide tools to help users compare how variation in parameters affects a given segmentation, and compare the result of two different segmentation techniques.
Variation in parameters. To segment the VOI of the presynaptic vesicle above, we used the continuous maximum-flow approach of Appleton et al., (2006). The segmentation results are influenced by the initial positions for the iterative schemes as defined by the rough polygonal outline of the object of interest, and the amount of segmentation regularization desired (i.e., how smooth a final segmentation outline is desired). Other than visual inspection ruling out obvious mis-segmentations) one cannot be sure of the resulting segmentation quality. ImageSurfer 2.0 will display the segmentation results for a range of parameter values. The figure below shows the segmentation results for different smoothness levels (ranging from 0: least smooth, to 1:most smooth). All segmentations were obtained from identical user input, specifying four seed points, which result in the turquoise polygon as an initial guess for the vesicle segmentation. Computing the segmentations for different smoothness settings reveals three main segmentation modes, one encompassing the main surface of the vesicles together with the small bump on the top left (depicted in white), a "yellow" mode, which omits the small bump and a mode that simply encloses the seed region (red). By providing interactive quantitative feedback, this approach enables the user to easily identify the most appropriate parameter settings.

A Segmentation results for varying levels of smoothness (0: least smooth, 1: smoothest). The turquoise region indicates the user polygon defined as an initial segmentation guess through the four starshaped seed points. B The yellow mode appears to be the most biologically plausible. C Contour length relative to the initial polygonal contour (as defined by the seed points). The segmentation is stable for three ranges of smoothness=[0,0.02] (white),  [0.03,0.15] (yellow), =[0.2,1] (red).

Display Surface Differences. Different segmentation algorithms produce different surfaces, and each will be different from an isosurface.  Within a given technique, different parameter settings can produce very different surfaces.  We intend to provide tools to enable the direct visual comparison between two such surfaces, to enable the scientist to easily compare such differences. Several tools will be provided, including tools base on principal curvature with point correspondence glyphs. The figure below illustrates this approach.

Example of the principal curvature with point-correspondence glyphs technique. Left panel: Comparison of an apple (hypothetically obtained by one segementation methods) and a pear (hypothetically obtained by another segmentation methods). The curved yellow lines reveal the mismatch between the two segmentations. Right panel: Comparison of a brain tumor segmented from MRI data by hand (blue) and an automatic algorithm (green).

We will add both interactive measurement and summary-statistic tools.

Interactive query tools used during initial exploration of the data will measure such things as vesicle diameter, distance between surfaces, angle of intersection of tubes, and the largest sphere that can easily move through a mesh. These tools will operate in both the full 3D space and in the user-positioned 2D slice plane. These tools are positioned and oriented in 3D by the user, based on the structures visible in the data set. We will augment these tools to also report variance data. The most complex of these tools, such as the spline tool shown in figure 1G (page 6), produce graphs of one or more values sampled along user-specified curves in the volume. The user can place curve specifying nodes either within the full 3D volume or within the 2D slice plane and obtain graphs of all data sets sampled along the curve.

Summary-statistic tools are used during the later stages of analysis, producing scalar summary values within volumes or surfaces. These include both single-variable measurements (volume, surface area, etc.) and multi-variable measurements (correlation, volume intersection) on both single and multiple objects (determined using connected-components tool) in the scene. These tools will all be implemented so that they can be used in combinations to produce more sophisticated queries.