Stephanie’s Note: Dr. Sena Sanjines just wrapped up her dissertation, part of which measured whether my Data Visualization Checklist is worth its salt. Here are her findings.

My name is Sena Sanjines and I’m an evaluator in Hawai‘i slightly obsessed with figuring out what makes people use, or not use, evaluation reports. Also, I love data visualizations – I can spend hours tweaking fonts, colors, lines, and text until a visualization sings with the takeaway finding. In the last few years though, something’s been bugging me. Even while more and more evaluators are getting into data viz, there still seems to be a reluctance among some to embrace graphic design in reporting which many see as lowering the legitimacy or rigor of reports. Also, on the flip side, I noticed many evaluators whole-heartedly embraced data visualizations and used them for everything, with all the bells and whistles, whether or not they helped communicate the data.

But the real problem is this, we just don’t have enough research to tell us if adding data visualizations to reports makes a difference or not. So last year, with the help of Stephanie Evergreen, I set about answering two questions: Does the use of data visualizations increase the likelihood a report will be used, and does the quality of data visualizations increase the likelihood a report will be used?

Check out Stephanie’s study this one was built on here.

I’m going to skip to the end of the story and tell you right off the bat that I didn’t find a relationship between data visualizations and use of reports, at least not within the funky politicized data I had access to for the study. (Check out my recent presentation on this here). But what I did find was this; The Data Visualization Checklist is a good measure of the quality of data visualizations and we can use it for research.

Also, I found that reports that were more like advocacy research, magazine-quality, with recommendations, etc., were more likely to be used than those that looked like traditional research (think peer-reviewed journal style). The main finding, though, was that more research is needed in this area.

Still interested? Read on…

A little known fact: The original Data Visualization Checklist developed in 2014 was created for evaluators so they could use it to make good data visualizations in their reports. Stephanie Evergreen and Ann Emery collaborated to make the checklist based on research and their own experience helping folks make better graphs. They revised it in 2016 to be more clear and to make it applicable to everyone, not just evaluators, so we could go from something like the graph below, used with permission from Brandon and Singh (2009, p.127)…

To something like this…

Very helpful indeed. The trick is, I wanted to use the Data Visualization Checklist for research, to measure the quality of the graphs in my study, and it was not created for that. This meant that whatever I found from using the checklist in my research may not be valid. Also, I had no clue if the checklist was reliable. For instance, if two people used it to rate the same graph, would they use it in the same way? Dunno.

One way to learn if the Data Visualization Checklist measured the quality of visualizations was to see if people understood and used it for that purpose. So what, did I just sit down and ask people how they used the checklist to rate a graph? …Exactly! It’s called a cognitive interview and I did nine of them. A cognitive interview is structured to get at what’s inside people’s heads: “Walk me through you were thinking while you rated that item…” I used the interviews to see if people’s understanding of the checklist aligned with the underlying research on cognition used to create it. I had each person rate a graph, recorded them, analyzed it, and found – yup! folks saw each guideline in the checklist aided either the readability or interpretability of the graph. Woo-hoo! This finding was in line with the original grounding of the Data Visualization Checklist and research on cognition and design.

The great thing is the interviews not only gave insight into how people understood the Data Visualization Checklist, it also highlighted parts of the checklist which gave everyone a hard time. I analyzed those too and found all were related to ambiguous language in the guidelines. So, I talked to Stephanie to make sure I understood the guidelines well and created a training to tell people exactly how to read each one and use them to rate a graph. A training on how to use the Checklist?! Where can I find such a thing? Here. While you’re at it, take Stephanie’s interactive Data Visualization Checklist for a ride.

Okay, evidence the Data Visualization Checklist measures data quality? Check! Next, I needed to see if it was reliable. Did people use it in the same way so we can trust scores from different raters? Time for the stats! A group of lovely humans volunteered to rate a ton of graphs for my study. Fourteen of those beautiful souls rated the same five graphs and I was able to compare their scores to check interrater reliability – that idea of people applying the checklist to rate graphs in roughly the same way. I used an Intraclass Correlation (ICC) and did a two-way consistency average measures ICC with mixed effects. What?! Basically, I was looking if, on average, a group of raters scored a random selection of graphs in the same way. The result was 0.87 which is considered good interrater reliability (Koo & Li, 2016) and basically means that 87% of the differences in how people scored the graphs were due to the checklist. In plain language, the Data Visualization Checklist (when used in combination with the rater training) is reliable.

The Data Visualization Checklist was not created for research. It was made so that you and I could use it to make our own graphs better (thank the heavens!). But wait!…There’s more! My research generated evidence that the checklist is also a solid measure of data visualization quality and not only that – it’s a reliable one.

This brings us back to the main point. Even though I didn’t find a direct connection between the use and quality of data visualizations and use of reports, I did find that reports more like advocacy research were used more and we don’t know exactly why that is. So, I’m closing with a challenge. You love data visualizations, that’s why you’re reading this blog. And I know you care, or believe that good visualizations make a difference, otherwise you wouldn’t bother making your graphs as good as they can be… But, a lot more research is needed.

Need a research idea? My crew of volunteer raters and I used the Data Visualization Checklist to rate over 1,000 graphs in hundreds of reports but what I didn’t write about in my findings was this weird thing I noticed: Most did not promote a take-away message, which seems like a main point of the checklist. So here is my new question for all of us: When making good graphs, are some design elements more important than others?

Email me your comments and questions about this study or your ideas for other research on data visualizations.

Brandon, P. R., & Singh, J. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation use. American Journal of Evaluation, 30(2), 123–157. http://doi.org/10.1177/1098214009334507

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. http://doi.org/10.1016/j.jcm.2016.02.012