*Updated Note from Stephanie: This blog post generated a lot of discussion. Some of that is in the comments here, some of that has been deleted, some of it came from Twitter and via my inbox. Be sure to read the comments to get a sense of the critique. At the bottom of this post, before the comments, I’ve provided some of my reasoning because I think context helps and to explain why I’ve had to delete comments (Hint: Threats!)*

*Note from Stephanie: I outlined a few ways to show regression data in my latest book but they all avoid the regression table itself. This guest post from William Faulkner, João Martinho, and Heather Muntzer illustrates how to improve the simple table and how to take that data even further into something that doesn’t require a PhD to interpret.*

**The world is going to hell in a handbag, and it’s because data viz people haven’t stepped up to the plate. **

There’s a dark horse of data viz hidden in plain sight, which has for decades made a mess of one of humanity’s most crucial quantitative tools. This villain is the regression table.

**The world runs on regressions.**

How many times have you heard “studies show that [blah],” or “it turns out [blah] leads to [blah]”? More often than not, these ‘facts’ are (over)simplified interpretations of a regression. Regressions are THE most common statistical way to determine whether there’s a relationship between two things – like doing yoga and wearing tight pants, or, as we’ll see in a sec, a person’s race and likelihood of being shot by the police.

**People interpret the results of regressions using regression tables (and little else)**

A ton of super important decisions get made on the basis of simple statements like “studies show you can reduce [blah] by [blah]%.” And these invariably come from a regression table, which usually looks something like this example analysis of 1974 cars, testing whether those with automatic or manual transmissions are more efficient:

(R user? DIY kit available here)

**Regression tables are TERRIBLE visualization tools. The WORST.**

Nobody wants to look at that thing! Are you kidding? And worse, even if you’re a quantitative genius really interested in the results, it’s STILL hard to intuit what’s going on.

**For The Love Of Humanity, Let’s Fix This. **

**WARNING: This middle section is for the nerds. If you don’t run regressions yourself, feel free to skip down to Section III**

### The Tame Tweaks

Even without going wild, we can just stop being so *careless*. With just a few simple tweaks, we can go from this:

To this:

We’re not claiming perfection, but at least we’re not being as cruel to our audience. Seems a lot easier now to see that the automatic-manual distinction is not as important for efficiency when we account for weight and horsepower.

### Let’s Go Nuts

Think outside the box (ahem, table), when it comes to regressions, maybe we can just graph the coefficients?

#### How changing the horsepower, weight, and transmission type of the average 1974 car seems to affect its mileage:

Again, not perfect. But it’s a start towards diagrams that intuitively __show__ what we really care about in most cases:

- Size of coefficients
- Uncertainty of coefficients (confidence intervals and/or statistical significance)
- Explanatory power of the model overall

And that’s it!

**Because It **__Is__ Important.

__Is__Important.

Last year, Harvard professor Dr. Fryer released a working paper inspiring some controversial headlines.

Did the paper really claim that blacks were 23% *less* likely than whites to be shot during an encounter with police? The whole hullabaloo boils down to – you guessed it – a regression table which is, as per usual, practically indecipherable:

Let’s try our tweaks from before:

Add a possible title, like:

#### As we factor in other variables, such as whether the suspect had a weapon, whether the bias is towards blacks or whites becomes a lot less clear __and__ we can’t be nearly as precise about the __amount__ of bias.

That’s better. Not perfect, but better. Turns out as we consider more and more aspects of the encounter, that strong bias towards police shooting white suspects gets a lot more muddled.

**And It’s Not Even That Hard. **

But you’re right. Who cares about nuance? In a world which constantly steamrolls detail in the name of thumbs-up or thumbs-down now-and-forever conclusions, who’s got time to worry about subtleties?

We’d like to think some people do. We’d like to think oh-so-many-more *would *take interest were it not for these bristling anathemas – regression tables. Regression analysis and data viz experts, let’s give folks a chance:

- Be nice to your audience. If you put a regular, white, asterix-splattered regression table in front of them, that’s inconsiderate. So don’t. Instead,
- Use accessible labels, translate jargon
- Take out extra decimals
- Use color, shading, and transparency to express the key info in multiple ways.

- Consider what’s important about the analysis – this means both the finding itself
*and the degree of uncertainty surrounding it*! Tools like heatmaps and the coefficient charts above help to put all that detail out there in a quickly digestible format.

Big problem. Reeeeeasonably easy solutions. Now the fate of the world is up to you. No pressure.

**Authors:**

William Faulkner – Director, Flux RME

João Martinho – Evaluation Specialist, C&A Foundation

Heather Muntzer – Independent Designer

PS: We asked for data from the Harvard team to replicate this study and produce even better visualizations. Despite a few kind replies, they never got around to sharing it.

PPS: More materials from this project are available in this Google Drive folder.

PPPS: Questions? Email William.

# Updated Editor’s Note:

I’m updating my editor’s note on this post because of how laughably out of hand things have gotten. Statisticians are really unnerved by some of the wording used in this guest post. It’s ok to disagree. I welcome those discussions and comments because they help everyone keep evolving their thinking.

But I’m heavily screening all comments posted to this thread from this point forward because now I’m getting threats like this one:

If you do not bring that uninformative guest post down, or fix it, I will bring this to the attention of the media and my fellow colleagues at ASA. Trust me, you do not want that kind of attention.

Um what? Report me to the American Statistical Association? LOL

More than one person took issue not with the content of the blog itself, but with the way that I asked the critics to improve upon what they didn’t like. One person wrote:

I am profoundly upset with your and Faulkner’s reaction to the comments.

Rule #1, as I’ve stated before, is that this is my blog. I’ll say what I want. I’ll outright and without apology delete any comments that attempt to tell me how to handle commenters or whether to pull a post. If you feel so strongly that it is bad, don’t read it. Start your own blog.

That doesn’t mean I defend errors. It means that I’m ok with mistakes – I’ve made plenty of my own – especially when they foster good discussion. But good discussion means generation of new ideas. Only one statistician in all of this mix has agreed to make a better attempt – in a few weeks.

Yes, of course, I’m asking critics to do better. Being an armchair critic is easy. To paraphrase Brene Brown, if you aren’t in the arena with me – actively trying to make things better by putting forth efforts that could be wrong or critiqued – I’m not interested in your opinion.

Several commenters questioned my intelligence. And then one guy (they were almost all white guys) said:

People are just trying to help you.

LOL you dudes are so funny. Insulting my intelligence is not help.

It might help, actually, to understand a bit more about me and the guest post authors. I have a PhD in interdisciplinary research and evaluation. I met the guest authors at an evaluation conference. (Did you see that the lead author’s name is William Faulkner?? How can you NOT have tequila with this guy?) If you don’t know what evaluation is, it’ll be good for me to explain it to you because there’s a pretty big difference between conducting pure statistics and evaluation.

Evaluators are like researchers in that we seek to generate knowledge but we conduct our studies for real organizations who are trying to learn whether they’ve made an impact with their work, or whether new strategies could help them be more efficient. We use anything from observation to a random controlled trial to get at the data. Our methods often have to be creative, since we are collecting data from actual humans, not in clinical settings. Our analyses are always rigorous. And we have to generate explanations of those analyses for real human decision-makers, in time for them to actually make use of it.

Our audience is real life, not a journal. Those explanations can be very challenging to compose. It can be difficult to balance statistical jargon fidelity with the need to speak in a plain language for the understanding and action-taking on the table with our clients. Will and team made one of the first attempts I’ve ever seen at making regression more digestible for people. Of course the first attempt will never be perfect. But kudos to them for giving it a shot, instead of just running some stats and wondering why the audience doesn’t get it (or worse, questioning the audience’s intelligence).

Twice as many people sent love and support for this post as those statisticians who got furious. And that’s because people are hungry and eager for something better than the way the stats people have been doing it. So keep building. Ever forward, friends.

Oh and please please PLEASE report me to the American Statistical Association! I’ve seen some slidedecks from their conferences over the years. You could use my help.

Ok, so I’m posting this at the request of Stephanie via twitter (in case it seems like I’m just wandering about the internet being a jerk). I also want to preface this with the fact that I agree with the fundamental premise of the post. Regression tables are ugly and not a great way of communicating results to non-technical audiences. Something better IS needed, so good on you for taking a swing at this.

“Short” form: The suggested visualizations propagate a fundamental misunderstanding of what p-values are. Please do not use them to communicate regression results, or suggest others do so.

They also use strong causal language when describing the meaning of the coefficients, which is not appropriate without further information. E.g., maybe fuel efficient engines are lighter than less efficient ones, thus meaning that actually it is an increase in efficiency that causes a decrease in weight, rather than the other way around. (I’m not suggesting this is the case, but it’s equally supported by the data).

Long form: The p-value is the probability of getting an apparent effect/relationship as-or-more extreme as the one we got on our data assuming the null hypothesis (in this case slope = 0). In frequentist statistics (the type you are doing when you call the lm function in R to get that table) it’s not meaningful to talk about probabilities of slopes doing things. We assume the slope has a fixed, but unknown value and are attempting to a) estimate it, and b) quantify how confident we are that our estimate is “close” to the truth. Even in Bayesian statistics, where probabilities about slopes are reasonable, however, we can’t talk ab out the probability of the slope being exactly 0 in most cases. This has to do with how probability works when the value can be anything in a continuous range of numbers (i.e. anything between 0 and 5, for example, or any real number).

Another even more important point is that we can never say the null is true. That isn’t how statistics works. That’s why we talk about things being “significantly different from 0” when we reject the null hypothesis, but only say that “there is insufficient evidence to conclude that the slope is significantly different from 0” if we don’t reject. This may sound like semantics but it’s actually pretty important for reasons related to how easy it is to manufacture a study that fails to reject if that’s your goal by intentionally designing it to be underpowered.

Can you construct a visual that you would feel better about? Please do so and share.

Definitely this. The point isn’t about the visualization but the language. Was gonna say the same thing about p-values and causality.

Can you make an attempt at better wording?

I think getting the wording right is one of the most challenging aspects of visualizing (and yes the language IS part of what makes a viz). These folks have given it a shot. Build upon it, don’t just tear them down.

Stephanie,

I’m not trying to tear anyone down (note I never referred to the authors of the post anywhere in my initial response), but sometimes things are simply not correct. It doesn’t mean the authors are bad or stupid people (they’re surely better at the aesthetics of viz than I am, for one thing) but it does mean they don’t understand what they’re trying to visualize in this case, and it lead to a visualization that misses the mark.

The issue is that so far as I know, the value 1-pvalue is not intended to be meaningful. It technically has a meaning, of course: the probability of getting an apparent relationship less extreme, i.e. slope closer to 0, than we observed when sampling from the null distribution. It’s related to the definition “confidence” used in confidence intervals, but I don’t think it should be talked about that way for reasons having to do with the technical definition of confidence and what confidence interval we would be talking about (it’s not the one we want).

As for what a better visualization is. Confidence intervals around the coefficient estimates are good (better than p-values and standard errors). Or if you only have a few variables you car about, make scatter plots of them individually against response, and overlay the multivariate regression line projected down onto that dimension (note this is different than the single variable regression line, in general). Just be sure when you talk about the slope of that line you use something along the lines of “controlling for all other variables in our model a one unit increase in is *associated with* a ____ unit (increase|decrease) in *on average*”

Best,

~G

You tickle me Stephanie! To laugh about MR is to feel joy!

For the love of humanity, please continue to fix things. 😉

It would be useful to put Stephanie’s comments (via Gabe) above the post so that readers can judge the statistical expertise of the writers prior to reading. I got to the point where they use a common but errant definition of p and stopped reading (luckily looked at the comments). It would be a shame if any readers left believing these basic misconceptions of the post.

I had assumed the post was going to deal with communicating the conditional nature of individual coefficients (instead it ignores it saying things like “every added half ton …”). That is often difficult in posts, but certainly some good work (e.g., related to the lasso) has been done.

I’d love to see your attempt at an improved visual. Please make one and share.

Sure. I’m away for a week and then will have a week or so to catch up, but then I’d send something to you.

Rad!

Rad it is! I’ll email something to you. Dan

Sorry if I was unclear. I made a comment on twitter and Stephanie asked that I comment on the blog. The above comments were mine, not Stephanie’s and I did not intend to indicate otherwise.

While I understand the need to avoid jargon and increase accessibility, if I received the “tweaked” tables for peer review I would have difficulty recommending them for publication. The reason we use regression coefficients/estimates and standard errors is because they have precise meaning and can be interpreted properly by those who know. I support the ideas of this post but think it might be best to put an effective visualisation in the paper/report and give a decent table with jargon in appendices/support material. Otherwise if the author has misinterpreted their result (e.g. The shooting data) it is very difficult to work backwards and see what the true result could/should be.

Also +1 on accurate interpretation of p-values. It’s really, really tricky to put a p-value in plain English without misinterpreting it. ‘Probability this result is observed due to a real effect rather than chance’ is still not great but the best I can do for now.

Thanks for the input. I don’t know that their target audience here is a place that would require peer review.

Dear Dave, Gabe, Caitlin,

This is not really a post about statistics and I apologize if somehow it came across that way. This post is more about the use and communication of statistics to wide audiences, an area where I hope we can agree the status quo is pretty far from optimal.

There are of course dangers to relaxing the rigor with which study results are communicated, but as it stands now, things tend to jump straight from massively complicated analyses with tables upon regression tables and jargon-studded interpretations to “studies show eating lettuce stresses your dog out” or “no bias in police shootings.”

To me, the risks of misinterpretation (and mis-use) are INCREASED by insistence on semantic exactness. As soon as you say “null hypothesis,” most people who might care about the results stop paying attention and stop wanting to understand any subtleties. Just watch how the Fryer study rippled out into the news! There is a middle ground between stiffly upholding the long-form, exact explanations and being so relaxed that all interpretations are groovy. The point is that data viz people need to be working HARD to build that bridge. It’s about getting it better, not getting it right.

In that vein, allow me to try again:

1. Regressions are super common but the way they are communicated beyond some very small circles is oversimplified to the point that all most people ever understand about results is “x does [not] cause y.”

2. One of the leverage points where things could easily be improved is the regression table – because it is so obviously a terrible data viz tool.

3. Let’s see if we can use data viz to communicate a little more nuance in less space, for example: (a) size of estimated effect, (b) uncertainty surrounding estimation, (c) explanatory power of the model overall.

4. And it’s not that hard to do better (no perfection required).

(a) Translate jargon. I stand by my interpretations, not because they are accurate representations of the statistical meaning, but because they might help a wider audience understand that there IS this thing called uncertainty around estimated effects. But I’m also positive it could be done better and I entice others to propose how.

(b) Take out extra decimals (use basic data viz principles & don’t repeat information – just trying to give specific examples).

(c) Tell a visual story – encourage people to rely less on wordy interpretations and more on the quantitative results themselves – and this means incorporating things like color, shading and transparency to communicate more in less space.

Like Stephanie, I am most interested in seeing what can be proposed – how we can do better than the status quo which creates such a huge gulf between the people doing regressions and the people using their results. A gulf – I might add – which makes it such that most of the users have to rely on the regressionists’ very human interpretations of the results, as expressed through ever-ambiguous human language.

So, whether this post and commentary represent a carrot or a stick, the point is things have to change, and the regression tables are an optimal place to start.

Just insert data viz.

Will

Absolutely – I may not have been clear communicating that I fully support this effort and that improvement is needed. I just think there is a role for regression tables in their more pure form as supplementary material so that the info being communicated can be checked for accuracy. But especially the repeating info and decimal point bits – always makes me cringe. I think a standard going forward is hard to achieve but efforts to improve need to be ongoing!

I particularly like the idea of viz as a graph rather than coercing the table into something “interpretable”.

Folks,

I think it’s definitely worthwhile trying to translate regression tables into more output that’s more accessible for users. But I think that it would have been a good idea to consult with someone with a good grasp of what regression results actually mean before publishing this advice. The interpretations given aren’t casual but reasonable translations of statistical jargon – they’re really just wrong (and *importantly* wrong, in that they can lead to bad real-life decisions).

-A p value is not the probability that the null hypothesis is true, and in many circumstances won’t even be close to that probability.

-A standard error of a coefficient is not the average distance of points from a regression line (you might be thinking of the standard error of the residuals, and even then the definition is suspect).

-A confidence interval does not indicate the “likelihood” that the true parameter falls within the bounds given

-An odds is not a likelihood or a probability

And that isn’t even touching on the subtler stuff like avoiding causal interpretations of regression coefficients, or communicating the idea that slopes are conditional on the other predictors being held constant.

Again… I appreciate the intent of the post, but I think there’s an underlying assumption here that regression is fundamentally easy to understand, and that the reason most regression tables *aren’t* easy to understand is simply that researchers can’t be bothered presenting them in a nice accessible way. That is *not* what is going on – regression is harder to understand than you think. It can be made easier to understand – e.g., by employing Bayesian estimation and thus achieving more intuitive statements about probability and uncertainty – but that requires actual analytic work on the part of the researcher (not just a bit of re-jigging of tables).

Yes you’re right – it’s really hard. And these folks have given it the first shot I’ve ever seen. They may not have gotten it perfect, but they started the conversation. So build on it. Do it better. Show your work.

Stephanie and guest authors – thanks for launching this discussion. You’re doing important work. Keep going.

As someone who isn’t a statistician, I found this blog post very informative. I often have to present regression results to fellow non-stats people and if I just tossed up a regression table, then no one understands what’s going on and I waste time explaining what each row/column mean…precious time lost on detail no one really cares in the end…and there is miscommunication in all directions, yawns, etc. And 99.99% of the time, non-stat people don’t care what the p-values are — they expect the presenter to already do all the work of simplifying the information for us — just tell us the take home message with easy to understand evidence that supports whatever claim you are making. Interested individuals will ask for more info afterwards but most do “blindly” trust experts.

Critiques of the blog post are clearly not interested in educating or informing non-experts but to be self serving in their ivory tower — and give a bad rep to academics and intellects.

since you asked for examples… A couple of years ago when I was looking to talk to my staff about improving readability of reports I came across article which had some cool regression graphics ideas: Kastellec & Leoni (2007) Using Graphs Instead of Tables in Political Science.

http://www.jstor.org/stable/20446574?seq=1#page_scan_tab_contents

What they have is more appropriate for a report context than a presentation, there are still some good things to ponder there.

Thanks for sharing this!