Don't Compare Averages

A strip chart shows all the individual numbers

So the rule is don't compare averages when you don't know what the actual distribution of the data looks like. How can you get a good picture of the data?

I'll start with the case above, when we don't have very many data points. Often the best way to go for this is a strip chart, which will show every data point in the different populations.

ggplot(sales, aes(name, d_revenue, color=name)) +
  geom_jitter(width=0.15, alpha = 0.4, size=5, show.legend=FALSE) +
  ylab(label = "revenue increase (%)") +
  geom_hline(yintercept = 0) +
  theme_grey(base_size=30)

With this chart we can now clearly see the lone high point for Bob, that most of his results are similar to Alice's worst results, and that Clara is far more consistent. This tells us far more than the earlier bar chart, but isn't really any harder to interpret.

You may then ask, how to plot this nice strip chart? Most people who want to plot some quick graphs use Excel, or some other spreadsheet. I don't know how easy it is to plot a strip chart in the average spreadsheet, as I'm not much of a spreadsheet user. Based on what I see in management presentations, it may be impossible, as I hardly ever see one. For my plotting I use R, which a frighteningly powerful statistics package, used by people who are familiar with phrases like “Kendall rank correlation coefficient” and “Mann-Whitney U test”. Despite this fearsome armory, however, it's pretty easy to dabble with the R system for simple data manipulation and graph plotting. It's developed by academics as open-source software, so you can download and use it without worrying about license costs and procurement bureaucracy. Unusually for the open-source world, it has excellent documentation and tutorials to learn how to use it. (If you're a Pythonista, there's also a fine range of Python libraries to do all these things, although I've not delved much into that territory.) If you're curious about R, I have a summary in the appendix of how I've learned what I know about it.

If you're interested in how I generate the various charts I show here, I've included a “show-code” disclosure after each chart which shows the commands to plot the chart. The sales dataframes used have two columns: name, and d_revenue.

What if we have a larger number of data points to consider? Imagine our trio are now rather more important, each handling a couple of hundred accounts. Their distributions still, however show the same basic characteristics, and we can see that from a new strip chart.

ggplot(large_sales, aes(name, value, color=name)) +
  geom_jitter(width=0.15, alpha = 0.4, size=2, show.legend=FALSE) +
  ylab(label = "revenue increase (%)") +
  geom_hline(yintercept = 0) +
  theme_grey(base_size=30)

One problem with the strip chart, however, is that we can't see the average. So we can't tell whether Bob's high values are enough to compensate for this general lower points. I can deal with this by plotting the mean point on the graph, in this case as a black diamond.

ggplot(large_sales, aes(name, value, color=name)) +
  geom_jitter(width=0.15, alpha = 0.4, size=2, show.legend=FALSE) +
  ylab(label = "revenue increase (%)") +
  geom_hline(yintercept = 0) +
  stat_summary(fun = "mean", size = 5, geom = "point", shape=18, color = 'black') +
  theme_grey(base_size=30)

So in this case Bob's mean is a bit less than the other two.

This shows that, even though I often disparage those who use means to compare groups, I don't think means are useless. My disdain is for those who only use means, or use them without examining the overall distribution. Some kind of average is often a useful element of a comparison, but more often than not, the median is actually the better central point to use since it holds up better to big outliers like Bob's. Whenever you see an “average”, you should always consider which is better: median or mean?

Often the reason median is such an under-used function is because our tooling doesn't encourage us to use it. SQL, the dominant database query language, comes with a built-in AVG function that computes the mean. If you want the median, however, you're usually doomed to googling some rather ugly algorithms, unless your database has the ability to load extension functions. 1 If some day I become supreme leader, I will decree that no platform can have a mean function unless they also supply a median.

Using histograms to see the shape of a distribution

While using a strip chart is a good way to get an immediate sense of what the data looks like, other charts can help us compare them in different ways. One thing I notice is that many people want to use The One Chart to show a particular set of data. But every kind of chart illuminates different features of a dataset, and it's wise to use several to get a sense of what the data may be telling us. Certainly this is true when I'm exploring data, trying to get a sense of what it's telling me. But even when it comes to communicating data, I'll use several charts so my readers can see different aspects of what the data is saying.

The histogram is a classic way of looking at a distribution. Here are histograms for the large dataset.

ggplot(large_sales, aes(value, fill=name)) +
  geom_histogram(binwidth = 1, boundary=0, show.legend=FALSE) +
  xlab(label = "revenue increase (%)") +
  scale_y_continuous(breaks = c(50,100)) +
  geom_vline(xintercept = 0) +
  theme_grey(base_size=30) +
 facet_wrap(~ name,ncol=1)

Histograms work really well at showing the shape of a single distribution. So it's easy to see that Alice's deals clump into two distinct blocks, while Clara's have a single block. Those shapes are somewhat easy to see from the strip chart too, but the histogram clarifies the shape.

A histogram shows only one group, but here I've shown several together to do the comparison. R has a special feature for this, which it refers to as faceted plots. These kind of “small multiples” (a term coined by Edward Tufte) can be very handy for comparisons. Fortunately R makes them easy to plot.

Another way to visualize the shapes of the distributions is a density plot, which I think of as a smooth curve of a histogram.

ggplot(large_sales, aes(value, color=name)) +
  geom_density(show.legend=FALSE) +
  geom_vline(xintercept = 0) +
  xlab(label = "revenue increase (%)") +
  scale_y_continuous(breaks = c(0.1)) +
  theme_grey(base_size=30) +
  facet_wrap(~ name,ncol=1)

The density scale on the y axis isn't very meaningful to me, so I tend to remove that scale from the plot - after all the key element of these are shapes of the distributions. In addition, since the density plot is easy to render as a line, I can plot all of them on a single graph.

ggplot(large_sales, aes(value, color=name)) +
  geom_density(size=2) +
  scale_y_continuous(breaks = NULL) +
  xlab(label = "revenue increase (%)") +
  geom_vline(xintercept = 0) +
  theme_grey(base_size=30) 

Histograms and density plots are more effective when there are more data points, they aren't so helpful when there's only a handful (as with the first example). A bar chart of counts is useful when there are only a few values, such as the 5-star ratings on review sites. A few years ago Amazon added such a chart for its reviews, which show the distribution in addition to the average score.

Boxplots work well with many comparisons

Histograms and density plots are a good way to compare different shapes of distributions, but once I get beyond a handful of graphs then they become difficult to compare. It's also useful to get a sense of commonly defined ranges and positions within the distribution. This is where the boxplot comes in handy.

ggplot(large_sales, aes(name, value, color=name)) +
  geom_boxplot(show.legend=FALSE) +
  ylab(label = "revenue increase (%)") +
  geom_hline(yintercept = 0) +
  theme_grey(base_size=30) 

The box plot focuses our attention on the middle range of the data, so that half the data points are within the box. Looking at the graph we can see more than half of Bob's accounts shrank and that his upper quartile is below Clara's lower quartile. We also see his cluster of hot accounts at the upper end of the graph.

The box plot works nicely with a couple of dozen items to compare, providing a good summary of what the underlying data looks like. Here's an example of this. I moved to London in 1983 and moved to Boston a decade later. Being British, I naturally think about how the weather compares in the two cities. So here is a chart showing comparing their daily high temperatures each month since 1983.

ggplot(temps, aes(month, high_temp, color=factor(city))) +
  ylab(label = "daily high temp (°C)") +
  theme_grey(base_size=20) +
  scale_x_discrete(labels=month.abb) +
  labs(color = NULL) +
  theme(legend.position = "bottom") +
  geom_boxplot()

This is an impressive chart, since it summarizes over 27,000 data points. I can see how the median temperatures are warmer in London during the winter, but cooler in the summer. But I can also see how the variations in each month compare. I can see that over a quarter of the time, Boston doesn't get over freezing in January. Boston's upper quartile is barely over London's lower quartile, clearly indicating how much colder it is in my new home. But I can also see there are occasions when Boston can be warmer in January than London ever is during that winter month.

The box plot does have a weakness, however, in that we can't see the exact shape of the data, just the commonly defined aggregate points. This may be an issue when comparing Alice and Clara, since we don't see the double-peak in Alice's distribution in the way that we do with histogram and density chart.

There are a couple of ways around this. One is that I can easily combine the box plot with the strip chart.

ggplot(large_sales, aes(name, value, color=name)) +
  geom_boxplot(show.legend=FALSE, outlier.shape = NA) +
  geom_jitter(width=0.15, alpha = 0.4, size=1, show.legend=FALSE) +
  ylab(label = "revenue increase (%)") +
  stat_summary(fun = "mean", size = 5, geom = "point", shape=18, color = 'black') +
  geom_hline(yintercept = 0) +
  theme_grey(base_size=30) 

This allows me to show both the underlying data, and the important aggregate values. In this plot I also included the black diamond that I used before to show the position of the mean. This is a good way to highlight cases like Bob where the mean and median are quite different.

Another approach is the violin plot, which draws a density plot into the sides of the boxes.

ggplot(large_sales, aes(name, value, color=name, fill=name)) +
  geom_violin(show.legend=FALSE, alpha = 0.5) +
  ylab(label = "revenue increase (%)") +
  geom_hline(yintercept = 0) +
  theme_grey(base_size=30) 

This has the advantage of showing the shape of the distribution clearly, so the double peak of Alice's performance stands right out. As with density plots, they only become effective with a larger number of points. For the sales example, I think I'd rather see the points in the box, but the trade-off changes if we have 27,000 temperature measurements.

ggplot(temps, aes(month, high_temp, fill=factor(city))) +
  ylab(label = "daily high temp (°C)") +
  theme_grey(base_size=20) +
  labs(fill = NULL) +
  scale_x_discrete(labels=month.abb) +
  theme(legend.position = "bottom") +
  geom_violin(color = NA) 

Here we can see that the violins do a great job of showing the shapes of the data for each month. But overall I find the box chart of this data more useful. It's often easier to compare by using significant signposts in the data, such as the medians and quartiles. This is another case where multiple plots play a role, at least while exploring the data. The box plot is usually the most useful, but it's worth at least a glance at a violin plot, just to see if reveals some quirky shape.

Summing Up

• Don't use just an average to compare groups unless you understand the underlying distribution.
• If someone shows you data with just an average ask: “what does the distribution look like?”
• If you're exploring how groups compare, use several different plots to explore their shape and how best to compare them.
• If asked for an “average”, check whether a mean or median is better.
• When presenting differences between groups, consider at least the charts I've shown here, don't be afraid to use more than one, and pick those that best illustrate the important features.
• above all: plot the distribution!