Do special techniques for surveying sensitive topics provide valid measurement? A validation design that detects false positives

The valid measurement of sensitive issues such as norm-violations or stigmatizing traits through self-reports in surveys is often problematic. Validation studies show, that a considerable share of respondents falsely deny sensitive behavior when asked about such in surveys. In the best case, sensitive behavior is then just underestimated using such data, in the worst case, conclusions about correlates and causes of the sensitive behavior in question are plain wrong. Despite this serious flaw, research in deviance, epidemiology, and many other areas relies heavily on self-report data. Finding ways to achieve a valid measurement of sensitive items is, therefore, very important.
Special sensitive question techniques that use some randomization procedure to introduce systematic noise into the response process such as the Randomized Response Technique (RRT, Warner 1965) and among its variants the recent Crosswise Model RRT (CM, Yu, Tian and Tang 2008) should generate more honest answers by granting respondents full response privacy. Full response privacy means, that there is no possibility to infer from a single respondent’s response her actual answer to the sensitive question. In turn, respondents are supposed to answer more honestly and the validity of the self-reports should increases. However, the RRT sometimes does not work as expected and evaluating whether particular implementations actually improve data validity is essential before their application in substantive surveys.
Different types of validation studies have examined whether particular techniques actually improve data validity, with varying results. Yet, most of these studies did not consider the possibility of false positives, i.e. that respondents are misclassified as having a sensitive trait even though they actually do not. Assuming that respondents only falsely deny but never falsely admit possessing a sensitive trait or behavior, higher prevalence estimates have typically been interpreted as more valid estimates. If false positives occur, however, conclusions drawn under this assumption might be misleading.
We present an easy-to-apply comparative validation design that is able to detect systematic false positives without the need for an individual-level validation criterion – which is often unavailable. This is achieved by introducing one or more zero-prevalence items among the sensitive items. If a sensitive question technique systematically leads to false positives, the estimates of the zero-prevalence items will be non-zero. Results from its application in a survey on “Organ donation and health” (N=1,686) showed that the most widely used crosswise-model RRT implementation produced false positives to a non-ignorable extent. This serious defect was not revealed by several previous validation studies that did not consider false positives – apparently a blind spot in past sensitive question research.