Artificial Intelligence (AI) is increasingly being used in a wide variety of medical imaging applications. Most of the focus, however, is on algorithm and scheme development, but this is only part of the picture. In order to have an impact on clinical decision making, workflow and patient care these AI tools must be evaluated using real-world cases and actual clinical providers that are expected to use them in routine care. The techniques used to conduct these types of studies are less well known in this field thus investigators need to be trained the proper study design and analysis methods.
This course will cover basic principles, techniques, and process for validating models developed using artificial intelligence (AI)/machine learning (ML) techniques. The primary goal of this course is to help the audience understand and apply fundamental principles related to designing, executing, and interpreting model evaluation studies. The course will be organized around two parts: analytic validation and clinical validation. In the first half, the audience will be exposed to approaches for performing a technical validation of a prediction model, including different study designs, appropriate statistical tests, metrics, dataset considerations, and decision curve analysis. The second half will cover the process of undertaking clinical validation that would address real-world use of models, regulatory and deployment issues. Topics include workflow integration, prospective clinical trials, reader impact studies, and regulatory approvals. Examples will focus on imaging-related models that are drawn from literature and the instructors’ personal experiences in prognostic modeling, computer-aided diagnosis, and imaging biomarker development.
- Describe the process of analytic and clinical validation, elucidating the steps involved and the considerations in designing an evaluation
- Explain basic concepts (e.g., training/testing/validation, cross-validation) and metrics (e.g., precision/recall) related to algorithm evaluation
- Choose the appropriate study design and metrics for comparing algorithms, depending on data, model, and objective
- Interpret various metrics to determine whether one algorithm is superior over others for a specific task
- Identify sources of potential biases that may influence model performance due to characteristics of the training or target populations
- Identify potential steps in moving beyond an algorithm paper to deploying AI/ML in clinical practice.
- Identify appropriate methods to assess the impact of AI techniques and tools on observer accuracy.