Segmentation using a Shape Model

Initial threshold-based segmentation

It is first necessary to segment the object as well as possible using a simple threshold. Use the standard thresholding options in the draw task page to do this. You will be left with a set of contours defining the surface, perhaps quite accurately in places, but less accurately elsewhere.

Loading a shape model

Now switch to the "Fit shape model" task. Load a shape model in the form of a .ply file. The header of the file must include at least two, and preferably three or more, comments of the form

comment FIT_MODEL_LM "cochlear apex" 2436
comment FIT_MODEL_LM "centre of oval window" 5567
comment FIT_MODEL_LM "p-a canal bifurcation" 6735
describing anatomical landmarks and the indices of the mesh vertices where they are located. These landmarks are used for a rough, initial alignment of the model to the data. If Stradview finds, in the same folder, a file with the same name but with suffix .ssm (statistical shape model), it will load that too. .ssm files may be generated using wxRegSurf: they describe the principal modes of variation across a cohort of surfaces. One of Stradview's nonrigid model-fitting algorithms allows the surface to deform only along the shape modes contained in the .ssm file. Cochlear implant .ply files are a special case containing only comments to specify the electrode positions and geometry.

Model initialization

The next step is to fit the model very roughly using (preferably) three or more landmarks. Select the "mark" tool and click in the review canvas on each landmark in turn. A prompt reminds you which landmark to define next. After all landmarks have been defined, click "Fit to landmarks". You will see the red surface moving to approximately the right position, though it may require an additional reflection. You can toggle between reflected and unreflected fits by pressing the "Fit to landmarks" button repeatedly.

Alternatively, you can manipulate the model into roughly the right position by holding down the ctrl key while clicking and dragging in the 3D window. The left and middle mouse buttons rotate the surface, while the right button translates the surface. The scroll wheel enlarges or shrinks the model, except for cochlear implants where it increases or descreases the insertion angle. Press r in the 3D window to toggle between reflected and unreflected versions of the surface.

If only two landmarks are available, then the initial landmark-based alignment is ill-posed, since the model's rotation around the line joining the two landmarks is unconstrained. You can manually adjust this rotation by holding down the shift and ctrl keys while clicking and dragging in the 3D window.

The rest of the process involves refining the fit using the contours defined in the first step. Select the appropriate object in the "Points from" menu and then click "Map points". You will see in the review canvas a lot of red points derived from the contours. We are going to refine the position of the surface so that it aligns as well as possible with the closest red points, using an iterative registration algorithm. Points that are currently closest to the surface are shown in bright red, other points in dark red.

Points are not coloured bright red if the surface normals of the model and the thresholded object are pointing in opposite directions. However, this requires correct inside/outside segmentation at the thresholding stage. If there has been an error, and points are not coloured bright red when they should be, set the "Match inside/outside" radio box to "in" and then "Map points" again. To disable normal matching altogether, set the radio box to "both" and then "Map points" again.

Iterative registration

Scroll through the segmented frames and edit the red points manually. You will need to add some points (use the "mark" tool) where the thresholded contours did not capture the surface correctly, and perhaps also delete some points (use the "erase" tool) where there are confusing red points on the wrong surface. The "Display compounding" slider allows you to see red points on nearby frames too, and delete them all with one sweep of the "erase" tool. At any time, you can refit the red surface to the current set of red points by pressing the "Start registration" button. For now, leave the "Nonrigid flexibility/Number of SSM modes" control at zero, and run 10 registration iterations at a time, before reviewing the result. With no nonrigid flexibility, the fit is rigid plus isotropic scaling (7 degrees of freedom).

The "Search range" slider sets how far away from the surface Stradview is prepared to look for a red point. Initially, when the fit is poor, you will need quite a large value so that the registration algorithm can discover the correct red points and converge on a good solution. Later, you can reduce the slider value so that only nearby red points affect the registration process: all the other (dark red) points are ignored.

The "Landmark weighting" slider controls the degree to which the manually defined landmarks contribute to the iterative registration. If this slider is set to zero, landmarks are ignored and Stradview aims to minimize only the distances between the model vertices and the closest red points. If this slider is set to a higher number, then the optimization considers also the distances between the manually placed landmarks and the corresponding points on the model.

When you have a decent fit, and a decent set of bright red points to fit to (dark red points are less problematic), you can refine the fit by introducing some nonrigid deformation. If you did not load a .ssm file, a locally affine deformation (LAD) is used, controlled by the "Nonrigid flexibility" slider. If you did load a .ssm file, you may choose between LAD and the statistical shape model (SSM), selecting the "Number of SSM modes" to use with the slider. Start with a low "Nonrigid flexibility" or "Number of SSM modes". You may then need to manually refine the red points before doing a final fit with a higher "Nonrigid flexibility" or a higher number of SSM modes. If the surface gets tangled as you edit the red points, you can reset the nonrigid deformation by selecting zero "Number of SSM modes" or "Nonrigid flexibility", and then refitting the model.

Mapping back to contours

When you are happy with the fit, you can either map the fitted surface back to the draw task page ("Map surface"), or generate a set of segmentation contours from the fitted model ("Map contours"). In both cases, use the "Contours to" menu to select the object which will receive the segmentation result (this could be the same one you selected in the "Points from" menu). When mapping contours, you can choose whether to insert contours onto every frame or every n frames, counting from the current frame. You can also select the accuracy with which the contours should follow the fitted surface. Finally, by selecting "Low", you can generate contours only on frames 0 to the current frame. Or, by selecting "High", you can generate contours only on frames from the current frame to the final frame. This may be useful when the model represents only part of the surface, and you want to merge the model segmentation in some frames with the thresholded segmentation in other frames.

If any further editing is required, this can be performed using the standard tools on the draw task page.

Interpolating sub-structures

Optionally, the shape model .ply file may refer to other surface meshes as follows:

comment FIT_MODEL_MS "reference structure" reference_structure.ply
comment FIT_MODEL_MS "sub-structure 1" substructure_1.ply
comment FIT_MODEL_MS "sub-structure 2" substructure_2.ply
comment FIT_MODEL_MS "sub-structure 3" substructure_3.ply
comment FIT_MODEL_MS "sub-structure 4" substructure_4.ply
...
In this paradigm, the referenced .ply files define surface meshes in a common coordinate system, and the reference_structure.ply mesh must be topologically equivalent to the shape model mesh. Then, pressing the "Interpolate sub-structure" button calculates the radial basis function (RBF) interpolant that maps the reference structure vertices onto the fitted shape model vertices, and applies the RBF to each of the sub-structure meshes in turn. The resulting, interpolated meshes are imported automatically into the object indicated by the "Contours to" menu and its immediate neighbours: e.g. if Object 11 is selected then, for the example above, the interpolated meshes will appear in Objects 11-15. From here, the interpolated meshes may be saved or edited, as required, using the standard tools on the draw task page.