Share Email Print

Proceedings Paper

Non-localization and localization ROC analyses using clinically based scoring
Author(s): Sophie Paquerault; Frank W. Samuelson; Kyle J. Myers; Robert C. Smith
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

We are investigating the potential for differences in study conclusions when assessing the estimated impact of a computer-aided detection (CAD) system on readers' performance. The data utilized in this investigation were derived from a multi-reader multi-case observer study involving one hundred mammographic background images to which fixed-size and fixed-intensity Gaussian signals were added, generating a low- and high-intensity signal sets. The study setting allowed CAD assessment in two situations: when CAD sensitivity was 1) superior or 2) lower than the average reader. Seven readers were asked to review each set in the unaided and CAD-aided reading modes, mark and rate their findings. Using this data, we studied the effect on study conclusion of three clinically-based receiver operating characteristic (ROC) scoring definitions. These scoring definitions included both location-specific and non-location-specific rules. The results showed agreement in the estimated impact of CAD on the overall reader performance. In the study setting where CAD sensitivity is superior to the average reader, the mean difference in AUC between the CAD-aided read and unaided read was 0.049 (95%CIs: -0.027; 0.130) for the image scoring definition that is based on non-location-specific rules, and 0.104 (95%CIs: 0.036; 0.174) and 0.090 (95%CIs: 0.031; 0.155) for image scoring definitions that are based on location-specific rules. The increases in AUC were statistically significant for the location-specific scoring definitions. It was further observed that the variance on these estimates was reduced when using the location-specific scoring definitions compared to that using a non-location-specific scoring definition. In the study setting where CAD sensitivity is equivalent or lower than the average reader, the mean differences in AUC are slightly above 0.01 for all image scoring definitions. These increases in AUC were not statistical significant for any of the image scoring definitions. The results on the variance analysis differed from those observed in the other study setting. This investigation furthers our understanding of the relationships between non-localization-specific and localization-specific ROC assessment methodologies and their relevance to clinical practice.

Paper Details

Date Published: 12 March 2009
PDF: 9 pages
Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630U (12 March 2009); doi: 10.1117/12.813761
Show Author Affiliations
Sophie Paquerault, U.S. Food and Drug Administration (United States)
Frank W. Samuelson, U.S. Food and Drug Administration (United States)
Kyle J. Myers, U.S. Food and Drug Administration (United States)
Robert C. Smith, U.S. Food and Drug Administration (United States)

Published in SPIE Proceedings Vol. 7263:
Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment
Berkman Sahiner; David J. Manning, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?