Duncan Green’s post on NGO advocacy makes a case that well-designed advocacy gets results – in this case, reform of the Social Subsidies Agency in Georgia. Yet the case study highlights a difficult question about advocacy in general – how can you tell if your advocacy has had any results, and how can you tell if the results of your advocacy have had the impact that you intended? To summarize the sequence of events as presented by Duncan (and please excuse the extended quote):
Following the ‘Rose Revolution’ of 2003 the new government made an attempt at revamping the country’s cumbersome and ineffective social protection scheme, inherited from the Soviet Union. In 2005, it set up the SSA (Social Subsidies Agency) to manage its new social assistance programme…The government and the donors declared victory, but monitoring by Oxfam and a local partner NGO – the Association of Young Economists of Georgia (AYEG) -showed that the system was not working… We held a number of closed door seminars with the representatives of the SSA (without involving the media so as to build trust) and backed them up with lobbying meetings. Nothing happened – we built up good relationships with young and motivated civil servants, but there was no appetite from their bosses for overhauling what was generally portrayed as a success. That all changed with the civil unrest of 2007, a political shock born of public disenchantment with government policy and reforms. The government was desperate to refurbish its image…Step forward a new and sympathetic Minister for Healthcare, with a background of working with NGOs, who gave his political backing…
Now this sounds (to me, at least) like quite a common story. It usually takes an external shock of some kind for any organisation to make radical changes – in this case, first the Rose Revolution, and then public protests against the path taken since the Revolution. The problem is that this account of events also sounds (again, to me) like a large number of biases stacking up to explain something that was largely out of the two NGOs control. Duncan reports that
… our Georgia team put success down to three factors: the shift in the political environment; the rigorous use of evidence and the cultivation of contacts with a range of decision makers and officials.
Now clearly the shift in the political environment was a major factor, but the other two factors? The case looks much weaker. Duncan has described the government as “desperate”, and I doubt that the rigor of the evidence that was presented to them was as critical as the fact that they were being offered a ready-made reform package that they could sell to the public. The “cultivation of contacts” were undoubtedly useful, but the key figure appears to have been a new Minister with an NGO background. This is crucial, because somebody from an NGO background is likely to have an automatic bias towards proposals from NGO (and the sort of approachs that NGOs propose). The Georgia team’s appraisal of what made change possible definitely has some merit – but it also looks like a way of retrospectively justifying personal and professional investment in an extremely worthwhile and challenging project.
Now I have no opinions on the actual reform proposals that Oxfam and AYEG put forward – certainly the brief description in Duncan’s article sounds good, but then the original concept of the SSA sounds pretty good as well. I have no reason to think that this reform has done anything except improve the welfare of the people the SSA is intended to support – but I also think that we need to examine the potential biases that lead us to assume that Change Y is the result of Action X, particularly in macro environments which are extremely complex.
This holds true for any area of aid work – how justified are our claims that our work helps people?