Interactive query tools used during initial exploration of the data will measure such things as vesicle diameter, distance between surfaces, angle of intersection of tubes, and the largest sphere that can easily move through a mesh. These tools will operate in both the full 3D space and in the user-positioned 2D slice plane. These tools are positioned and oriented in 3D by the user, based on the structures visible in the data set. We will augment these tools to also report variance data. The most complex of these tools, such as the spline tool shown in figure 1G (page 6), produce graphs of one or more values sampled along user-specified curves in the volume. The user can place curve specifying nodes either within the full 3D volume or within the 2D slice plane and obtain graphs of all data sets sampled along the curve.

Summary-statistic tools are used during the later stages of analysis, producing scalar summary values within volumes or surfaces. These include both single-variable measurements (volume, surface area, etc.) and multi-variable measurements (correlation, volume intersection) on both single and multiple objects (determined using connected-components tool) in the scene. These tools will all be implemented so that they can be used in combinations to produce more sophisticated queries.

A Segmentation results for varying levels of smoothness (0: least smooth, 1: smoothest). The turquoise region indicates the user polygon defined as an initial segmentation guess through the four starshaped seed points. B The yellow mode appears to be the most biologically plausible. C Contour length relative to the initial polygonal contour (as defined by the seed points). The segmentation is stable for three ranges of smoothness=[0,0.02] (white),  [0.03,0.15] (yellow), =[0.2,1] (red).

Display Surface Differences. Different segmentation algorithms produce different surfaces, and each will be different from an isosurface.  Within a given technique, different parameter settings can produce very different surfaces.  We intend to provide tools to enable the direct visual comparison between two such surfaces, to enable the scientist to easily compare such differences. Several tools will be provided, including tools base on principal curvature with point correspondence glyphs. The figure below illustrates this approach.

Example of the principal curvature with point-correspondence glyphs technique. Left panel: Comparison of an apple (hypothetically obtained by one segementation methods) and a pear (hypothetically obtained by another segmentation methods). The curved yellow lines reveal the mismatch between the two segmentations. Right panel: Comparison of a brain tumor segmented from MRI data by hand (blue) and an automatic algorithm (green).

Flow chart of image processing and analysis using ImageSurfer 2.0. After preprocessing of an image using deconvolution and image filtering, objects of interest are defined on the image. Segmentation algorithms are based on the volume of interest definition. Upon inspecting the output of the segmentation algorithm and its parameter sensitivity, segmentation settings are refined, in an iterative manner: subsequently new volumes of interest can be defined based on the segmentation results, allowing for refined object extraction and  ultimately, quantification of image information.

The figure below shows how our two-stage approach performs at segmenting a presynaptic vesicle from electron tomographic material. The user only has to specify a few "seed" points within the vesicle on one single slice. The algorithm then automatically finds the 3D vesicle boundaries, defining the VOI. Tests confirm that the result is robust to seed placement, as long as the seeds are placed strictly inside the vesicle. Segmentation is then obtained automatically through three-dimensional seeded region-growing, where the seed areas were obtained by watershed segmentation performed on the two-dimensional manifold described by the vesicle surface, instead of directly in voxel space.

Semi-automatic segmentation of a presynaptic vesicle. A, raw image. B, The user places seed points in the interior of the vesicle. C, D The vesicle is automatically detected in 3D space, defined by its membrane. E, 3D watershed segmentation is performed, defining individual proteins or proteins cluster. F, G 3D views of the segmented objects.

We will make available:

1. Point-spread-function (PSF) specification/estimation using four distinct methods:

a. Analytic specification of the PSF: Analytic PSF models will be provided, based on known microscope parameters (i.e. numerical aperture, media index of refraction, wavelength, voxel size).
b. Empirical estimation of the PSF: Estimation of the PSF through measurements of microscopic beads. Averaging capabilities for multiple measurements (to increase signal to noise ratio) will be provided.
c. Parametric-Empirical estimation of the PSF: Analytic specification of the PSF relies on an accurate parameterization of the PSF, empirical estimation on accurate measurements. To combine the benefits of both methods, we will implement a maximum-likelihood estimator to identify the parameters of the analytical model based on the empirical measurements of the PSF, effectively combining approaches #a and #b above.
d. PSF Estimation through blind deconvolution: We will implement blind deconvolution methods to provide an alternative data-driven method to estimate a PSF without the need for explicit measurements of the microscope's impulse response. In addition, we will provide non-parametric estimation of the PSF (based on bandwidth constraints and symmetry assumptions) as well as parametric estimation, analogous to (c)).

2. Non-blind deconvolution algorithms, which use the specified or estimated PSF from (1).

3. Blind deconvolution algorithms, which combine PSF estimation and image deconvolution.

Contextual help. Tooltips appearing on the screen when the user moves his mouse over an interface element will provide the user with information about the element he is currently interested in. In addition, we will provide a hint (by displaying a special icon in the tooltip) that more documentation is available for that element. Thus, if the user finds that the information provided in the tooltip is not sufficient, he/she is pointed to the next level of help.

Support for more microscope-specific formats. Due to the large number of microscope-specific file formats, loading datasets into ImageSurfer can be difficult for novice users. ImageSurfer already supports several microscope-specific formats; we will expand this further.

Support 3D-specific devices. Interacting with complex 3D representations is inhibited by the two dimensional nature of interaction and display hardware. 3D images are usually viewed on flat screens; the sense of 3D is conveyed by a variety of features, including occlusion, perspective, and shading based on directional illumination. However, this does not produce the same impression achieved when we look around a room; that sense of depth also requires binocular disparity. The required software support for stereo viewing is built into the VTK toolkit that ImageSurfer uses for display. We will support a variety of stereo techniques (anaglyphic red/cyan, frame-sequential, or line-interleaved) using consumer-grade hardware.

User-friendly updater. Users will be able retrieve and install updates directly through an updater within ImageSurfer. The updater will also obtain and update plug-in software. It will display available plug-ins along with a description of their function; with a single click, users will be able to get more information, install or update a plug-in.

Diagram of ImageSurfer 2.0 architecture. ImageSurfer 2.0 will be composed of custom plug-ins and application code running on top of the existing OverView/ParaView/VTK architecture. External developers will be able to write plug-ins using the same interface (or a Python-based interface) to augment ImageSurfer with new algorithms and techniques. Other C++-based libraries such as μManager can be linked through plug-ins.

The Roadmap has three main components:

1. Strengthen ImageSurfer by developing a new core.
2. Make ImageSurfer more easily usable.
3. Provide new tools for image restoration and analysis.