Share Email Print

Proceedings Paper

Rating scales for observer performance studies
Author(s): Robert M. Nishikawa; Yulei Jiang; Charles E. Metz
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

We compared the performance of radiologists reading a set of screening mammograms with and without CADe as measured by the BI-RADS assessment scale to that measured by a 9-point rating scale. Eight MQSA radiologists read 300 screening mammograms, of which 66 cases contained at least one cancer and 234 were normal based on two-year follow-up. Both without and then with CADe, the radiologists gave their BI-RADS assessment for each case and, for each suspicious lesion in the image, reported their confidence on a 9-point scale (1=no evidence for recall; 5=equivocal; 9=overwhelming evidence for recall) that the lesion needed to be worked up. The radiologists were instructed to read the cases as they would clinically. We used MRMC ROC analysis employing PROPROC curve fitting to analyze the data, once for the BI-RADS data and again for that collected on the 9-point scale. Given that the radiologists were reading screening mammograms and were instructed to read in their normal clinical manner, not all radiologists used the full BI-RADS scale. Two radiologists used only BI-RADS 0,1 and 2, three used the full scale, and three used the full scale but employed categories 3, 4 and 5 sparingly. This mimics what occurs clinically, according to the literature. The BI-RADS and the 9-point rating scales gave similar results in terms of AUC. However, the 95% CIs of the estimates of AUC were substantially smaller for the 9-point scale.

Paper Details

Date Published: 4 March 2010
PDF: 7 pages
Proc. SPIE 7627, Medical Imaging 2010: Image Perception, Observer Performance, and Technology Assessment, 762703 (4 March 2010); doi: 10.1117/12.844993
Show Author Affiliations
Robert M. Nishikawa, The Univ. of Chicago (United States)
Yulei Jiang, The Univ. of Chicago (United States)
Charles E. Metz, The Univ. of Chicago (United States)

Published in SPIE Proceedings Vol. 7627:
Medical Imaging 2010: Image Perception, Observer Performance, and Technology Assessment
David J. Manning; Craig K. Abbey, Editor(s)

© SPIE. Terms of Use
Back to Top