Share Email Print

Proceedings Paper

Analyzing ROC curves using the effective set-size model
Author(s): Frank W. Samuelson; Craig K. Abbey; Xin He
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

The Effective Set-Size model has been used to describe uncertainty in various signal detection experiments. The model regards images as if they were an effective number (M*) of searchable locations, where the observer treats each location as a location-known-exactly detection task with signals having average detectability d'. The model assumes a rational observer behaves as if he searches an effective number of independent locations and follows signal detection theory at each location. Thus the location-known-exactly detectability (d') and the effective number of independent locations M* fully characterize search performance. In this model the image rating in a single-response task is assumed to be the maximum response that the observer would assign to these many locations. The model has been used by a number of other researchers, and is well corroborated. We examine this model as a way of differentiating imaging tasks that radiologists perform. Tasks involving more searching or location uncertainty may have higher estimated M* values. In this work we applied the Effective Set-Size model to a number of medical imaging data sets. The data sets include radiologists reading screening and diagnostic mammography with and without computer-aided diagnosis (CAD), and breast tomosynthesis. We developed an algorithm to fit the model parameters using two-sample maximum-likelihood ordinal regression, similar to the classic bi-normal model. The resulting model ROC curves are rational and fit the observed data well. We find that the distributions of M* and d' differ significantly among these data sets, and differ between pairs of imaging systems within studies. For example, on average tomosynthesis increased readers’ d' values, while CAD reduced the M* parameters. We demonstrate that the model parameters M* and d' are correlated. We conclude that the Effective Set-Size model may be a useful way of differentiating location uncertainty from the diagnostic uncertainty in medical imaging tasks.

Paper Details

Date Published: 7 March 2018
PDF: 8 pages
Proc. SPIE 10577, Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment, 105770E (7 March 2018); doi: 10.1117/12.2294529
Show Author Affiliations
Frank W. Samuelson, U.S. Food and Drug Administration (United States)
Craig K. Abbey, Univ. of California, Santa Barbara (United States)
Xin He, U.S. Food and Drug Administration (United States)

Published in SPIE Proceedings Vol. 10577:
Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment
Robert M. Nishikawa; Frank W. Samuelson, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?