A novel approach has been developed in this thesis for the problem of segmenting volumetric layers, a type of structure often encountered in medical image analysis. This approach is aimed towards the use of structural information to enhance the performance of the segmentation process. While some organs have more consistent global shape and can be characterized using a specific shape model, other anatomical structures possess much more complex shape with possibly high variability which needs a more generic shape constraint. The three-dimensional (3D) nature of anatomical structures necessitates the use of volumetric approaches that exploit complete spatial information and therefore are far superior to the non-optimal and often-biased 2D methods. Our method takes a volumetric approach, and incorporates a generic shape constraint---in particular, a thickness constraint. The resulting coupled surfaces algorithm with a level set implementation not only offers segmentation with the advantages of minimal user interaction, robustness to initialization and computational efficiency, but also facilitates the extraction and measurement of many geometric features of the volumetric layer.
The algorithm was applied to 3D Magnetic Resonance (MR) brain images for skull-stripping, cortex segmentation and various feature measurements including cortical surface shape and cortical thickness. Validation of the model was done through both synthetic images with "ground truth" and a wide range of real MR images with expert tracing results. As a natural follow up to the segmentation work, a new approach was developed for the extraction of sulcal ribbon surfaces which axe distinctive cortical landmarks of the brain. This effective and efficient 3D method of sulcal ribbon extraction has potential in a variety of applications such as the automatic parcellation of cortical regions and the problem of geometry-constrained brain atlas building. The tools of cortical and sulcal shape analysis developed in this work are of great importance to studies of neuroanatomy through medical imaging, and axe bringing about new understanding of brain anatomy and function.