How survey question phrasing greatly affects results

In a BBC television series episode titled The Grand Design Yes, British Prime Minister Jim Hacker tried to enforce conscription; his private secretary, Bernard Woolley, commissioned a poll that revealed British support for the concept. However, Woolley was shown by Sir Humphrey Appleby, the Cabinet Secretary, how a sequence of leading questions and a certain way of phrasing the key subject might lead to contradicting results in public polls.

About forty years after the series premiered, in February of this year, market research company Ipsos conducted an interesting study in the United Kingdom. 2,158 British residents between the ages of 16 and 75 were included in the representative quota sample. Of those questioned, half answered “Sample A” questions, indicating that they had a favorable opinion of national service.

The other half had a poor opinion of “Sample B.” A analysis of the data showed that answers to questions on the level of support for the reinstatement of national service in Britain varied depending on the question’s phrasing and the questions that came before it. In Sample A, 45% supported the reinstatement of conscription, while 38% opposed it. In Sample B, on the other hand, 38% agreed with it and 48% disagreed.

Writing the questionnaire is one of the most crucial phases in making sure a survey is successful. Understanding people’s thought processes and response patterns is crucial to designing a questionnaire that circumvents the biases and limitations of human cognition and produces pertinent, trustworthy findings.

Single questions are very important in psychology, but they are also quite controversial. The fascinating case given by Israeli-American psychologist and behavioral economics pioneer Daniel Kahneman was a fictitious 41-year-old lady called Linda who studied philosophy, took part in anti-nuclear rallies, and was politically active. To what extent does she resemble a bank teller?

Or how probable is it that she works as a bank teller and is involved in the feminist movement?” questioned Kahneman, the 2002 Nobel laureate in economics. Ninety percent of the respondents claimed she was more likely to be a bank teller who identified as feminist, which contradicts logic. But the first choice required her to be a bank teller in order to be a feminist bank teller. However, most individuals will be misled if the question is asked in this way. Therefore, it’s important to consider both the question you ask and how you express it.

Take another example from Britain in the 1980s. In a Gallup survey conducted in 1986, 40% of participants stated they felt “safe” knowing that nuclear weapons were in place, while 50% said they didn’t (the remaining respondents were unsure). When a separate pollster used the term “safer” instead of “safe,” the numbers shifted: 50% of respondents indicated they felt safer as a result of nuclear weapons, while 36% said they felt less safe.

Gallup conducted a study in June 1969 to gauge public opinion in the US over the “faster” or “slower” withdrawal of soldiers from Vietnam in response to then-President Richard Nixon’s directive to remove 25,000 troops in three months. The interviewers accepted responses from respondents stating “same as now,” even though it was not an option. Regarding the responses, 42% said it was “faster,” 16% said it was “slower,” and 29% said it was “same as now.”

A comparable survey, conducted immediately after the Gallup poll, asked respondents, “In general, do you feel the pace at which the President is withdrawing troops is too fast, too slow, or about right?” Harris asked. In response, 49% of participants believed the pace was about right, 29% wanted it to end sooner, and 6% said it was moving too quickly. But why are these two surveys different from one another? They were different, however, since Gallup omitted a clear intermediate group.

In reality, people’s responses to gains or privileges vary from those to losses or restrictions. The term “framing” gained popularity thanks to Kahneman and Amos Tversky. They modeled a hypothetical epidemic of an uncommon Asian illness that was expected to kill 600 people in a 1981 book. There were two different strategies proposed to contain the pandemic. Program A would save 200 lives if it were put into action. 600 persons had a one-third probability of being rescued in Program B, and a two-thirds chance of not being.

According to their study, 72% of participants were risk-averse and would rather save 200 lives rather than take a chance on saving more. A follow-up survey was conducted to choose between Programs C and D, which had the same results but different descriptions. 400 people would die if Program C was implemented. There was a two-thirds likelihood that someone would die from Programme D and a one-third chance that nobody would. Seventy-eight percent of the respondents said they would rather gamble than accept the definite loss of 400 lives, even if C and D were quantitatively identical to A and B.

In an attempt to identify the reasons behind this kind of uncertainty, Kahneman wrote Thinking, Fast and Slow in 2011. He explained how the fusion of two different systems resulted in people’s mental processes. “System 1,” or the gut response, is “quick, intuitive, and emotional.” The description of “System 2” is “slower, more deliberative, and more logical.” While System 1 operates automatically and without conscious thought, which usually kicks in first, System 2 requires work and focus. System 2 is only turned on when it is thought to be required.

Bernard, there you are. The entirely well-balanced sample,” Sir Humphrey said, going on to illustrate how disparate question styles may lead to inconsistent survey findings. People rarely consider the following, though, when they come across the results of a public opinion survey: were the questions filtered? Were there alternatives with a “middle category”? Were the questions phrased in terms of gains or losses? How were the questions posed? Which questions came before which? And other related questions? Sadly, the majority of people blindly believe that this is Sir Humphrey’s “perfectly balanced sample.”