Methodology deep dive đ€
Hello from SurveyMonkey! In this weekâs newsletter, weâre taking you behind the scenes to share some of the methodological considerations we think about when weâre writing a questionnaire. First, weâll share some new research findings we learned while attending a conference for public opinion researchers last week. Then, weâll walk through the results of a quick survey experiment we fielded recently.Â
Â
ThreeđĄ research topics we were tracking at AAPOR Â
How to ask about gender. Demographic questionsâespecially genderâare often sensitive topics for respondents, and understanding how to ask them in a way respondents can understand and appreciate is important. Read more from SurveyMonkey on SOGI questions here.Â
Survey data quality. Whatâs the best way to judge whether a survey is high quality? The usual tools researchers use include looking for speeders (people who complete a survey too quickly), straightliners (people who give the same answer to several questions in a row), and respondents who fail various âtrap questionâ tests embedded in a survey. Check out SurveyMonkeyâs recommendations for data quality here.Â
Probability vs. non-probability surveys. Most people using SurveyMonkey will be based on a non-probability sampling methodology. As non-probability surveys become ever more prevalent, methodologists (including our Research team!) are continuously exploring new ways to calibrate the results from non-probability and probability samples. Learn more about nonprobability sampling here.Â
Why question wording mattersÂ
In the spirit of sharing survey best practices, weâre dedicating this weekâs newsletter to the importance of question wording. As The Pew Research Center notes, the choice of words and phrases when writing surveys is critical: âeven small wording differences can substantially affect the answers people provide.â Proper question wording should allow the respondent to answer with reliability, i.e. if asked the same question again, they should give the same response.Â
In a recent SurveyMonkey poll, we ran an experiment to test just how much question wording matters. If you read our newsletter last week, youâll recall that we asked respondents to rate the favorability of four tech billionaires: Elon Musk, Jeff Bezos, Mark Zuckerberg, and Bill Gates. We found that Muskâs favorability had increased over the prior two weeks, while he has been in the news regularly for his (possible) takeover of Twitter.Â
In the experiment, a randomly selected half of the respondents were asked the series of favorability questions with each personâs title (i.e. âDo you have a favorable or unfavorable opinion of Elon Musk, CEO of Tesla and SpaceXâ), while the other half of respondents were asked the same favorability questions without the detail of who each person is. Â
In almost every case, respondents were more likely to say they âdonât have enough information to sayâ how they would rate someone when the personâs title was not included. Â
For Bezos, changes in unfamiliarity were particularly noticeable without his title: 26% didnât have enough info to say with his title included in the question, yet without his title, this percentage jumps to 35%.Â
Bezosâs favorability rating decreased too: without his title, Bezos had a favorability rating of -20 (21% favorable, 41% unfavorable, 35% donât know). With his title, Bezosâs favorability rating decreased 4 points to -24 (29% favorable, 43% unfavorable, 26% donât know).Â
Similarly, without including Zuckerbergâs title in the question, adults were less familiar with the Facebook founder: 27% of adults didnât know enough to say compared with 21% when Zuckerbergâs title was included. Respondents were also slightly more likely to have an unfavorable opinion when his title was included than without his title (59% vs. 54%).  Â
Differences werenât quite as pronounced for Musk: with his title included in the question, 23% didnât have enough info to say whether they favor him or not, but without his title, this percentage increases slightly to 26%.Â
Gates was the only exception: there was no real change in the percentage of those who donât know him with his title included in the question, 23% didnât know enough to say; without his title, 24% didnât know enough to say. So, in comparison to Musk, Bezos, and Zuckerberg, the majority of adults are familiar enough with Bill Gates to have an opinion, with or without his title. Interestingly, with Gatesâs title included, adults were slightly more likely to view him as favorable than without his title (39% vs. 35%).Â
Thatâs all for this week! Thanks as always for reading.Â