Naldi, LuigiLuigiNaldiBettoli, VincenzoVincenzoBettoliSantoro, EugenioEugenioSantoroValetto, Maria RosaMaria RosaValettoBolzon, AnnaAnnaBolzonCassalia, FortunatoFortunatoCassaliaCazzaniga, SimoneSimoneCazzaniga0000-0001-8161-6138Cima, SergioSergioCimaDanese, AndreaAndreaDaneseEmendi, SilviaSilviaEmendiPonzano, MonicaMonicaPonzanoScarpa, NicolettaNicolettaScarpaDri, PietroPietroDri2025-03-062025-03-062024-11-28https://boris-portal.unibe.ch/handle/20.500.12422/205833The large language model (LLM) ChatGPT can answer open-ended and complex questions, but its accuracy in providing reliable medical information requires a careful assessment. As part of the AICHECK (Artificial Intelligence for CME Health E-learning Contents and Knowledge) Study, aimed at evaluating the potential of ChatGPT in continuous medical education (CME), we compared ChatGPT-generated educational contents to the recommendations of the National Institute for Health and Care Excellence (NICE) guidelines on acne vulgaris. ChatGPT version 4 was exposed to a 23-item questionnaire developed by an experienced dermatologist. A panel of five dermatologists rated the answers positively in terms of "quality" (87.8%), "readability" (94.8%), "accuracy" (75.7%), "thoroughness" (85.2%), and "consistency" with guidelines (76.8%). The references provided by ChatGPT obtained positive ratings for "pertinence" (94.6%), "relevance" (91.2%), and "update" (62.3%). The internal reproducibility was adequate both for answers (93.5%) and references (67.4%). Answers related to issues of uncertainty and/or controversy in the scientific community scored the lowest. This study underscores the need to develop rigorous evaluation criteria for AI-generated medical content and for expert oversight to ensure accuracy and guideline adherence.en600 - Technology::610 - Medicine & healtharticle10.48620/857973996905810.4081/dr.2024.10138