I still have a landline. I’m not an old, but I conduct a lot of webinars about data visualization and reporting and the sound quality is so much better on a landline. But it is a midterm election year which means my phone has been ringing several times every day with calls from poll and opinion firms wanting to ask me questions about my voting intentions. I wish they all had read Designing Quality Survey Questions by Sheila B. Robinson and Kimberly Firth Leonard.
Why? Because Sheila and Kim fundamentally come from a place that respects the survey respondent. My favorite quote in the book:
We take to heart the notion that a survey is a form of conversation and social exchange and that respondents are rarely, if ever, compelled to answer questions for us. This drives us to use a “respondent-centered” design.
Amen. And then they spend the rest of the book telling you exactly how to do it.
If you have been steeped in survey research for most of your adult life, like me with a PhD in interdisciplinary research and evaluation, some of the book’s content may seem familiar. Even still, I will always chuckle at a triple-barreled question so I welcomed the review.
But its been a decade since I took my survey classes and, while Sheila and Kim cited some of my long-time favorite studies, times have changed. This book has so much juice in it!
Three cheers for…
The strong recommendation to only ask questions you have a clear plan to use, along with solid recommendations on what questions to cut or consolidate. Yes please! Don’t ask about my gender if you don’t need that information.
And if you do, this book has thoughtfully addressed how to word survey questions related to sensitive topics like race, ethnicity, gender, income, and religion. We did not address this enough when I was in grad school.
The few times open-ended questions make sense and how to introduce them so people will actually fill them out.
Whether to have a midpoint in your Likert scale (I’m team NO).
How better questions can help us avoid a “don’t know” response.
Clear, detailed, and realistic discussion of what respondents tend to think when presented with common survey issues.
Survey design considerations for respondents who have low-vision or blindness.
Fun ways to encourage responses.
And the narrative is accompanied by smile-worthy cartoons from the one and only, Chris Lysy, as well as adorable icons that make me jealous I didn’t have those in my book.
I’m cheering so hard because anyone who runs in the survey world needs to read and integrate this stuff and the other good ideas in this book.
What I Missed
I so wanted to see Sheila and Kim expand on their initial discussion of sampling. I know, I know. That’s a whole different book. I’m just frustrated by seeing folks put surveys out on Twitter (a convenience sample) and draw conclusions about an entire population (“People who do data visualization are most likely to write code.”) when they can really only draw conclusions about the tiny subpopulation that took the survey.
Sheila and Kim can’t save me from every frustrating thing about being a survey respondent but this book is so good I’m sure I’ll hear it reflected when I answer the phone in the future.
My favorite way to show Likert scale responses (withOUT a midpoint) is here. If you do end up with a lot of Don’t Know or missing data, here are some options for ya. Check this page for resources on how to handle those open-ended questions.
