Category Archives: M&E

The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases.

Josiah Charles Stamp was many things during his lifetime, including President of the Royal Statistical Society between 1930-32, and his view on government statistics is well worth bearing in mind whenever we look at the sort of statistics that tend to crop up in humanitarian and development work. International organisations tend to act in much the same way as governments when it comes to statistics, which we usually refer to as indicators.

Until very recently, all the information we had during a disaster was based on eyes on the ground – from residents in or visitors to affected areas. With the advent of remote sensing, we have a new source of information that doesn’t rely on actual presence – but even so, it’s worth noting that remote sensing without some form of ground truthing is frankly useless. I remember when we were looking at crop patterns in Afghanistan – an expert could pick out opium crops from a satellite image, but it was still necessary to send people to verify (and get shot at, of course – all part of the fun). So we still rely on eyes on the ground, which means that there’s always a human factor involved in data collection.

Where there’s a human factor, there’s always the scope for creativity that Stamp noted, or for deliberate manipulation. On the micro scale, that’s unlikely to make a huge difference, since when you aggregate up to a national level many of the irregularities will be levelled out – unless everybody at the micro level is fiddling the numbers in the same way. For example, anybody affected by a disaster is likely to exaggerate their needs if they think it will mean more assistance; any organisation responding to a disaster is likely to accept those exaggerations if it means they are likely to get more funding. As always, it’s good advice to follow the money.

So what does this mean for the poor information manager, tasked by his bosses to tell them what’s going on? Well, Stamp wasn’t saying that statistics were useless, only that we need to remember where they come from, which is rule number one: scrutinise your sources. He wasn’t saying that the village headman is out to cheat you, only that the village headman is human, so rule number two is: minimise the ways in which errors can be introduced to your data collection. As I noted above, those errors tend to get levelled out when you aggregate up – or at least to get less obvious amongst the mass of data – so rule number three is: where possible, always cross check your data against other sources.

Rule number four, of course, is: don’t expect statistics to solve all your problems. We tend to get a little fixated with a fetish for figures in this field (and I’m as guilty of that as anybody – look at the pretty pictures!) but, if our analysis or presentation aren’t solid, those figures aren’t going to be much use to managers.