Inter-Rater Reliability and Usability of CATHIS core for Homeopathic Intervention Studies.
Publisher DOI
PubMed ID
41765150
Abstract
Background
The Critical Appraisal Tool for Homeopathic Intervention Studies (CATHIS) core is a streamlined appraisal tool for homeopathic intervention studies focusing on credibility, coherence, and clinical relevance. The aim of the research project was to evaluate its inter-rater reliability, feasibility, and face validity.Methods
In a preregistered cross-sectional study, four raters independently applied CATHIS core to 28 trials (21 randomised controlled trials, 7 non-randomised studies on interventions) drawn from reviews on insomnia and hypertension; two external reviewers provided consensus ratings. Inter-rater reliability (IRR) was estimated using percent agreement, Fleiss' κ, and Gwet's AC2 (95% CIs). Feasibility was quantified as rating time and consensus time. Associations among the three domains were explored with correlation analyses and sensitivity checks.Results
IRR varied markedly by domain. Credibility showed good agreement (Fleiss' κ=0.66, 95% CI 0.57-0.74; AC2=0.76, 0.71-0.82). Coherence yielded only poor-to-fair agreement (κ=0.28, 0.16-0.40; AC2=0.41, 0.30-0.51). Clinical relevance was similarly limited (κ=0.32, 0.23-0.41; AC2=0.36, 0.28-0.44). Individual ratings required on average 65.8minutes, while consensus discussions averaged 17.7minutes. Correlation analyses indicated heterogeneous and partly overlapping domain signals with limited interpretability. Face-validity responses reflected moderate-to-high acceptance but difficulties in consistent application.Conclusion
CATHIS core yielded reproducible credibility ratings but only fair and operationally fragile agreement for coherence and clinical relevance, alongside non-trivial rating burden. Taken together, the current reliability profile is insufficient for confident use in systematic reviews. Targeted refinement appears warranted before broader implementation.
The Critical Appraisal Tool for Homeopathic Intervention Studies (CATHIS) core is a streamlined appraisal tool for homeopathic intervention studies focusing on credibility, coherence, and clinical relevance. The aim of the research project was to evaluate its inter-rater reliability, feasibility, and face validity.Methods
In a preregistered cross-sectional study, four raters independently applied CATHIS core to 28 trials (21 randomised controlled trials, 7 non-randomised studies on interventions) drawn from reviews on insomnia and hypertension; two external reviewers provided consensus ratings. Inter-rater reliability (IRR) was estimated using percent agreement, Fleiss' κ, and Gwet's AC2 (95% CIs). Feasibility was quantified as rating time and consensus time. Associations among the three domains were explored with correlation analyses and sensitivity checks.Results
IRR varied markedly by domain. Credibility showed good agreement (Fleiss' κ=0.66, 95% CI 0.57-0.74; AC2=0.76, 0.71-0.82). Coherence yielded only poor-to-fair agreement (κ=0.28, 0.16-0.40; AC2=0.41, 0.30-0.51). Clinical relevance was similarly limited (κ=0.32, 0.23-0.41; AC2=0.36, 0.28-0.44). Individual ratings required on average 65.8minutes, while consensus discussions averaged 17.7minutes. Correlation analyses indicated heterogeneous and partly overlapping domain signals with limited interpretability. Face-validity responses reflected moderate-to-high acceptance but difficulties in consistent application.Conclusion
CATHIS core yielded reproducible credibility ratings but only fair and operationally fragile agreement for coherence and clinical relevance, alongside non-trivial rating burden. Taken together, the current reliability profile is insufficient for confident use in systematic reviews. Targeted refinement appears warranted before broader implementation.
Date Issued
2026-05
Publication Type
Article
Subjects
Interrater reliability
•
clinical relevance
•
homeopathy
•
model validity
•
systematic reviews
Language(s)
en
Author(s)
Loef, Martin | |
Weiermayer, Petra | |
Gaertner, Katharina | |
Wrzałko, Daniel | |
Saha, Subhranil | |
Dutta, Abhijit | |
Lakshani, J A D S | |
van Haselen, Robbert |
Journal
Complementary Therapies in Medicine
Publisher
Elsevier
ISSN
1873-6963
0965-2299
Access(Rights)
open.access