Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use.

Abstract

Importance The primary objective of any newly developed medical device using artificial intelligence (AI) is to ensure its safe and effective use in broader clinical practice.Objective To evaluate key characteristics of AI-enabled medical devices approved by the US Food and Drug Administration (FDA) that are relevant to their clinical generalizability and are reported in the public domain.Design, Setting, And Participants This cross-sectional study collected information on all AI-enabled medical devices that received FDA approval and were listed on the FDA website as of August 31, 2024.Main Outcomes And Measures For each AI-enabled medical device, detailed information and key characteristics relevant for the generalizability of the devices at the time of approval were summarized, specifically examining clinical evaluation aspects, such as the presence and design of clinical performance studies, availability of discriminatory performance metrics, and age- and sex-specific data.Results In total, 903 FDA-approved AI-enabled medical devices were analyzed, most of which became available in the last decade. The devices primarily related to the specialties of radiology (692 devices [76.6.%]), cardiovascular medicine (91 devices [10.1%]), and neurology (29 devices [3.2%]). Most devices were software only (664 devices [73.5%]), and only 6 devices (0.7%) were implantable. Detailed descriptions of development were absent from most publicly provided summaries. Clinical performance studies were reported for 505 devices (55.9%), while 218 devices (24.1%) explicitly stated no performance studies were conducted. Retrospective study designs were most common (193 studies [38.2%]), with only 41 studies (8.1%) being prospective and 12 studies (2.4%) randomized. Discriminatory performance metrics were reported in 200 of the available summaries (sensitivity: 183 devices [36.2%]; specificity: 176 devices [34.9%]; area under the curve: 82 devices [16.2%]). Among clinical studies, less than one-third provided sex-specific data (145 studies [28.7%]), and only 117 studies (23.2%) addressed age-related subgroups.Conclusions And Relevance In this cross-sectional study, clinical performance studies at the time of approval were reported for approximately half of AI-enabled medical devices, yet the information was often insufficient for a comprehensive assessment of their clinical generalizability, emphasizing the need for ongoing monitoring and regular re-evaluation to identify and address unexpected performance changes during broader use.