• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain

Dolphin

Senior Member
Messages
17,567
This article was co-written by renegade psychologist, James C. Coyne.

I'm not highlighting it for any specific relevance to ME/CFS. It is just an example of a type of problem that can occur in questionnaires which I thought might be interesting to the odd person to highlight.

The paper is summarised in the first part of this blogpost: http://jcoynester.wordpress.com/201...-constructing-a-title-for-a-scientific-paper/


Abstract

Objective

We examined the effectiveness of reverse worded items as a means of reducing or preventing response bias. We first distinguished between several types of response bias that are often confused in literature. We next developed arguments why reversing items is probably never a good way to address response bias. We proposed testing whether reverse wording affects response bias with item-level data from the Multidimensional Fatigue Inventory (MFI-20), an instrument that contains reversed worded items.

Methods

With data from 700 respondents, we compared scores on items that were similar with respect either to content or to direction of wording. Psychometric properties of sets of these items worded in the same direction were compared with sets consisting of both straightforward and reversed worded items.

Results

We did not find evidence that ten reverse-worded items prevented response bias. Instead, the data suggest scores were contaminated by respondent inattention and confusion.

Conclusions

Using twenty items, balanced for scoring direction, to assess fatigue did not prevent respondents from inattentive or acquiescent answering. Rather, fewer mistakes are made with a 10-item instrument with items posed in the same direction. Such a format is preferable for both epidemiological and clinical studies.


Citation: Sonderen Ev, Sanderman R, Coyne JC (2013) Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain. PLoS ONE 8(7): e68967. doi:10.1371/journal.pone.0068967

The paper requires a bit of attention. However, I imagine it would be quite a bit less work to read if one skipped the Methods and Results sections.

Here's an extract from the introduction:

Response Bias

The process of obtaining survey data is complex, with many possibilities of discrepancies arising between the state or opinion the researcher wants to elicit and the answer given by the respondent [3]. Usually this discrepancy is called response bias [4]. Based on Rorer [1], Weijters [5] distinguishes two main types of response bias: response set and response style. He defines response set as bias related to the content of the items and response style as a tendency to answer items regardless their content. The best known type of response set is social desirability, in which a person’s response is a function of the desirability of the response rather than its veracity. Three types of response styles can be distinguished.

Respondents may have read and understood completely the question and answer categories, but nonetheless be inclined to agree with statements in general (acquiescence), to disagree (disacquiescence), or to give extreme answers, or, alternatively, less extreme answers. Consistent with Swain et al. [6], we will take acquiescence as an example of this type of response styles.

Respondents may also lack sufficient attention to carefully read both the question and answer categories, and thus, by missing the intended meaning of an item, give a response that may differ from the true value. Krosnick [7] mentions a satisficing response style, whereby the respondent deliberately makes less effort to understand all subtleties of the question. We will call this style inattention, irrespective whether the respondent is aware of it or not.

Finally, the question in combination with answer categories may be too difficult for a respondent to comprehend. Sometimes the respondent may think the item is well understood, but still an error can be made, due to a high level of difficulty [6]. The respondent may however be aware of this difficulty, and thus the answer can be considered a ‘best guess’. We will call both varieties of this type confusion.

I once observed some questionnaire responses from ME/CFS patients (unpublished study) which looked like there were some errors in the responses due to inattention. I will now have a name and reference for this sort of problem.
 

Dolphin

Senior Member
Messages
17,567
In the discussion section, they make the following point, which is to do with questionnaires/surveys in general rather than specifically what they were studying. I have seen user9876 bring up these sorts of issues.


Even when a multi-item questionnaire consists of items stated in the same direction, there are problems to be addressed that hamper an obvious relationship between the theoretical concept and the sumscore resulting from an addition of the itemscores [3]. Some aspects, commonly seen in multi-item instruments, that deserve to be addressed are:

1) Differences in item difficulty and their consequences for the interpretation of summed scores, a field that Item Response Theory is addressing.

2) Sometimes in the same questionnaire some aspects are addressed with more items than others, leading to an often unknown and implicit weighing of their contribution to the total score.

3) The rationale and consequences of using different answercategories for items that are supposed to belong to the same scale

4) The rationale and consequences of using both items asking for frequency and items asking for intensity.

All these phenomena deserve to be addressed. This discussion will be more fruitful if it is not obscured by effects resulting from reversed worded items.
 

Dolphin

Senior Member
Messages
17,567
Incidentally, the MFI-20 itself is used sometimes in ME/CFS research. For example, it is one of the questionnaires used in the empiric criteria (Reeves et al., 2005) for CFS. However, this was just a sample questionnaire they used.
 

user9876

Senior Member
Messages
4,556
Item response theory looks interesting but I haven't got round to doing more that a quick scan of it.

Another point that might not have been mentioned (I've not read the paper) which is around question ordering. I've come across at least one psychologist who recommended randomly choosing different question orders for each question to avoid leading the subject to an answer.

Negative questions are much harder to comprehend. I seem to remember some psycholinguistic results around the speed of processing negative sentences was greater based on monitoring eye movements as people scan the page. But would fit in with the "questions too hard" part.

Tversky and Kalman also found an effect where they got different decisions by structuring sentences to stress risk or reward - which perhaps demonstrates the importance of thinking through wording. Such work may be particularly relavent to some of the psychosocial researchers attempts to find causation that supports there models as they ask guestions around perceived risk of activity.
 

Dolphin

Senior Member
Messages
17,567
Another point that might not have been mentioned (I've not read the paper) which is around question ordering. I've come across at least one psychologist who recommended randomly choosing different question orders for each question to avoid leading the subject to an answer.

[..]

Tversky and Kalman also found an effect where they got different decisions by structuring sentences to stress risk or reward - which perhaps demonstrates the importance of thinking through wording. Such work may be particularly relavent to some of the psychosocial researchers attempts to find causation that supports there models as they ask guestions around perceived risk of activity.

They didn't discuss either of these that I recall but could imagine they might be aware of such issues (although am not sure).