Foveated Models of Visual Search and Medical Image Perception

Modern 3D medical imaging modalities such as computed tomography and digital breast tomosynthesis require that the radiologist scroll through a stack of 2D images (slices) to search for abnormalities (signals) predictive of disease. Here, I review the use of foveated computational models to understand various perceptual effects with radiologists. A foveated model processes the images with reduced fidelity from the fixation point, makes eye movements to explore the image, scrolls through images, and integrates sensory evidence across fixations and slices to reach decisions about the presence of signals. I show how the model predicts and explains a number of important findings with observers and radiologists. Small signals that are difficult to see in the periphery are often missed during search with 3D images. A synthetic 2D image created by combining all 3D slices can be presented as a preview to radiologists to guide their search through the 3D image stack and minimize the search errors of small signals. Variations in observers’ eye movements when searching for small signals in 3D images can fully account for variability in observer search performance. I will finalize by discussing recent efforts to integrate foveated models with deep q-learning techniques to estimate near-optimal (performance maximizing) eye movements during search with medical images.