Correcting crowdsourcing in a crisis
I hope that Patrick doesn’t feel that this is a hit post, because it isn’t intended to be. However I find it difficult to stay in my seat when I see the words “crowdsourcing” and “humanitarian” in the same sentence. So deep breaths, everybody. Patrick has just put up two posts, entitled Internews, Ushahidi and communication in crisis and Ushahidi: From Croudsourcing to Crowdfeeding. The posts were based on his attendance at a couple of events hosted in New York by InterNews (good organisation, by the way, we like them).
At the meetings, Patrick “introduced the concept of crowdsourcing crisis information” and says that he “should have expected the immediate push back on the issue of data validation”. These two posts taken together explain his position very well, but also explain why I am so very very unconvinced by that position – note that this doesn’t mean that I’m supportive of the status quo, which I’m not. More after the break:
Nobody noted the importance of timely information during crises, which is rather ironic since both meetings focused on sudden onset emergencies. I suspect that our demand (and partial Western obsession) for fully accurate information has clouded some of our thinking on this issue.
Both accuracy and timeliness are core Principles of Humanitarian Information management established at the 2002 Symposium on Best Practices in Humanitarian Information Exchange and reiterated at the 2007 Global Symposium +5. Have those principles been incorporated into the institutions sufficiently? Short answer, no. Is accuracy privileged at the expense of timeliness? Not in the field.
Field-based organizations frequently make decisions on coordination, humanitarian relief and logistics without complete and fully accurate, real-time information, especially right after a crisis strikes. So why is this same community holding crowdsourcing to a higher standard?
Because crowdsourcing is unfamiliar, it’s untested in the field and it makes fairly large claims that are not well backed by substantial evidence. Having said that, I’m willing to be corrected on this criticism, but I think it’s fair to say that the humanitarian community is legitimately cautious in introducing new concepts when lives are at stake.
I would also note that I don’t think the community is necessarily holding crowdsourcing to a higher standard, but exactly the same standard as our usual information systems – and if they haven’t managed to get those systems right yet, I can understand still further why they’re cautious about entertaining an entirely new and untested approach.
The figure below represents the perishable nature of crisis information. Data validation makes sense during time-periods A and B. Continuing to carry out data validation beyond time B may be beneficial to us, but hardly to crisis affected communities. We may very well have the luxury of time. Not so for at-risk communities.
This is where I get hazy – what “crisis information” are we talking about here? I would argue that ensuring that your data is valid is important at all times, so is this an attack on dissemination strategies rather than data validation? It also strikes me as odd that Patrick argues that affected communities deserve timely information but not necessarily accurate information. As he notes, it may be a trade-off – but he provides no argument for why he privileges timeliness over accuracy.
Ushahidi was created within days of the Kenyan elections in 2007 because both the government and national media were seriously under-reporting widespread human rights violations.
Does this mean that the government and national media have an obligation to report on absolutely every single violation of human rights taking place in their country? I’ll be honest, that’s not something I’d find either useful or interesting, and I’m not sure how it’s relevant to the humanitarian community. It might be relevant to the human rights community – but then you run into the validation question again, hard – and it still wouldn’t remove the need to validate that data.
This left the Ushahidi team with few options, which is why they decided to develop a transparent platform that would allow Kenyans to report directly, thereby circumventing the government, media and NGOs, who were working against transparency.
I’m not sure it’s a constructive approach to accuse NGOs of actively “working against transparency” – it strikes me that there may be some shades of grey in their attitudes towards releasing information about human rights abuses.
Here’s a question: why didn’t the human rights or humanitarian community set up a platform like Ushahidi? Why were a few tech-savvy Kenyans without a humanitarian background able to set up and deploy the platform within a week and not the humanitarian community?
I think it might be because the human rights and humanitarian communities were working on their existing projects. The argument that these organisations failed to fulfill an objective when they never actually had that objective in the first place is distinctly shakey – it seems to translate into a protest that they weren’t doing what you wanted them to do.
Having worked in humanitarian settings, we all know that accuracy is more often luxury than reality, particularly right after a crisis strikes… Rather, we need to start thinking in terms of likelihood, i.e., how likely is this piece of information to be accurate?
This, I can completely agree with; in fact I’ve been advocating for since 2000, with very little luck. The military does it much better than us, although I’m pleased to report that they’re nowhere near as good at it as we think they are (and they like us to think they are). However I would qualify that – perhaps we also have to start thinking in terms of what damage might this information do (whether true or false) if we release it.
The purpose of today’s meetings was for us to think more concretely about communication in crises from the perspective of at-risk communities. Yet, as soon as I mentioned crowdsourcing the discussion became about our own demand for fully accurate information with no concerns raised about the importance of timely information for crisis-affected communities.
I think we should consider that resistance to crowdsourcing may not be well-articulated but it is well-founded. The objections raised by the participants at the InterNews meeting included “scenarios whereby militias in the DRC would submit false reports to Ushahidi in order to scare villagers (who would receive the alert by SMS) and have them flee in their direction where they would ambush them.” Patrick thinks that their concerns are wrong, and lists the reasons why:
First of all, militias do not need Ushahidi to scare or ambush at-risk communities. In fact, using a platform like Ushahidi would be tactically inefficient and would require more coordinating on their part.
Well, they don’t need Ushahidi to achieve this, but evidence suggests that armed groups are perfectly happy to use whatever means they can acquire to achieve their goals. I fail to see why Ushahidi would be “tactically inefficient, and would require more co-ordinating” – all they need to do is send a few text messages. The entire point of the platform is that it’s easy to use, isn’t it?
The government of Sudan regularly accuses NGOs (well, those NGOs it hasn’t expelled) of misreporting human rights violations. What better tool would the government have for discrediting human rights monitoring than Ushahidi? All it would take would be a few texts a day with false but credible reports, and the government can dismiss the entire system, either by keeping their own involvement covert and claiming that the system is actually being abused, or by revealing their involvement and claiming that the system can be so easily gamed that it isn’t credible.
Second, local communities are rarely dependent on a single source of information. They have their own trusted social and kinship networks, which they can draw on to validate information. There are local community radios and some of these allow listeners to call in or text in with information and/or questions. Ushahidi doesn’t exist in an information vacuum. We need to understand information communication as an ecosystem.
That’s very true, but it’s not an argument for crowdsourcing.
Third, Ushahidi makes it clear that the information is crowdsourced and hence not automatically validated. Beneficiaries are not dumb; they can perfectly well understand that SMS alerts are simply alerts and not confirmed reports.
The idea that people pay attention to disclaimers makes me laugh out loud. I don’t think anybody’s accusing affected individuals of being dumb, but I’d be interested to see evidence that supports this claim. When does the validation take place, incidentally? And what recourse do individuals or communities have if an alert turns out to be false?
Fourth, many of the humanitarian organizations participating in today’s meetings work directly with beneficiaries in conflict zones. Shouldn’t they take an interest in the crowdsourced information and take advantage of being in the field to validate said information?
Yes, because they don’t have enough to do. They’d like to spend their time running around validating other people’s reports, endangering their lives and alienating the government under which they’re working.
Fifth, all the humanitarian organizations present during today’s meetings embraced the need for two-way, community-generated information and social media. Yet these same organizations fold there arms and revert to a one-way communication mindset when the issue of crowdsourcing comes up. They forget that they too can generate information in response to rumors and thus counter-act misinformation as soon as it spreads. If the US Embassy can do this in Madagascar using Twitter, why can’t humanitarian organizations do the equivalent?
Is there any evidence at all that the US Embassy’s Twitter feed had any impact at all on the course of events? I mean, I know it made a good headline in external media, but I don’t see how it’s a good example if there’s no actual evidence that it had any impact.
Sixth… Ushahidi deployments are localized and the crowdsourcing is limited to trusted members of those organizations, or journalists in the case of Al-Jazeera.
That doesn’t sound like crowdsourcing – that sounds like situation reporting. So what are we actually discussing here?
![[del.icio.us]](http://images.del.icio.us/static/img/delicious.small.gif){kind=small}
Important discussion – I can’t claim to have the answers, but I have one observation:
It might be very useful to the individuals suffering those human rights violations, and to other potential victims, if this kind of crowdsourced reporting turns out to have a deterrent effect.
Chris Watkins
30 Mar 09 at 16:52
@Chris Watkins: “It might be very useful to the individuals suffering those human rights violations, and to other potential victims, if this kind of crowdsourced reporting turns out to have a deterrent effect.”
Not every incidence of violence is a violation of human rights. I agree that bringing to light information about abuse is a necessary first step, but it only has a deterrent effect if there are consequences stemming from it. Accuracy is central to this, and without it there is no deterrent.
Making information about violence public has strong social and political mobilisation opportunities, which can have political consequences, but when people pin human rights labels to this information, though, they’re stepping into a very legalistic area with its own rules and strategies. To call something a human rights violation is to evaluate, and make a legal judgement about the nature of some source material. It is mainly this judgement that leads to consequences, such as an investigation of an allegation of abuse; as it’s this fear of judgement that is the most legitimate deterrent, the process of making the judgement has to be rigorous. Every time something is inaccurately called a human rights violation, it waters down the legal deterrent, because even the grunts on the ground know it has no real consequence for them.
However, it’s often unclear what constitutes a violation of human rights. This translates to argument about whether or not certain conduct is covered by the legal meaning, or whether the legal meaning should be expanded to cover the conduct. Even where the sources look clear cut, it can be far from the case. Watch this 5 minute clip from 30 July 2006:
http://www.youtube.com/watch?v=VfCjotmhERk
At first glance, at about 2m10s it shows three police constables from Sheffield (a big city in the UK) tackling and holding down a young black woman called Toni Comer, whilst a third punches her in the face and kicks her. The visual impact is stunning and very disturbing. After allegations made by Comer in the UK media about racism and brutality, the police officer involved – PC Anthony Mulhall – was suspended pending investigation. The subsequent Independent Police Complaints Commission (IPCC) inquiry concluded that despite the video footage, “the evidence collated during the investigation did not substantiate the allegation that the officers used unnecessary force” and there were no ground for criminal prosecution.
Where limited resource existed for the authorities, an independent institution or a civil society organisation to investigate, this case would certainly have been recorded incorrectly as yet another incidence of racism and police brutality. If it had been immediately rebroadcast during more tense times, in a more politically volatile environment, would it really have helped anyone? In my view, whatever the short term mobilisation opportunities, this sort of wilful inaccuracy harms the cause of social justice in the longer run.
Tom Longley
31 Mar 09 at 10:18
[...] Correcting crowdsourcing in a crisis [...]
Paul Currion on the “crisis” of crowdsourcing in a crisis «
31 Mar 09 at 13:51
Paul,
Great discussion points. We hope Patrick responds and will be directing readers to the discussion from our blog.
Noah
31 Mar 09 at 13:59
Hi Paul,
Just to note that not everything generated from the field, and subsequently verified to the extent possible, is for decision support in the present context. Much of what I and others are able to gather today from the field in Sri Lanka (and information from the ground is precious and dangerous to produce) is bearing witness, silently, at what is going on.
Used for parochial optics and propaganda, the value of crowdsourcing soon diminishes. This is not to say that information cannot be used howsoever their handlers choose to. I am merely trying to respond to your question as to whether a Twitter feed can shape events. Perhaps not the one you point to (but then again, who knows?) but the technology certainly can. Used to bear witness, cognisant that there is no one truth and that a multiplicity of viewpoints is better than fewer, technologies such as Ushahidi, Twitter and many others help a great deal in my line of work – which is self-effacing and outside the domain of headlines and handshake moments.
Making victims witnesses is possible through new technologies. We are just discovering how.
Warmly,
Sanjana
Sanjana Hattotuwa
31 Mar 09 at 14:53
Paul — solid points. Looking forward to continued dialog around how to best work with and improve the nimble, energetic, and increasingly well-known players like Ushahidi.
Paul G
Paul
31 Mar 09 at 15:36
@Sanjana – It would be quite interesting to map out better the life-cycle of information uses by progressive groups/orgs doing your kind of work (CPA?), and see which tools and approaches fit best where, with thier various ups and downs. There are far evermore new mechanisms for getting information, but ever fewer to help make sense of it, and organise it in ways that systematically serve the many uses it could be put to immediately and in the future. It’s how these systems work in concert that I find most interesting.
Tom Longley
31 Mar 09 at 16:23
Tom: the Toni Comer case illustrates a related point. In a celebrity-driven culture like the UK, making a guest appearance on YouTube is something a lot of people wouldn’t mind about. However I have questions about how much detail we want to see posted to the web – for example in cases where sexual violence is involved. Even if there is a clear human rights violation, if it involves (for example) rape, I doubt people would want to see it appearing on a Google Map; if there was enough level of detail to identify the specific case, it would be massively embarrassing, and if there wasn’t enough detail, then how useful would the information be?
Sanjana: The general principle of collecting detailed information about human rights violations and making them available in some useful form is not controversial. What bothers me is whether a) “crowdsourcing” as a methodology yields more information benefits than costs and b) how useful this information actually is in the form(s) that it will be collected. Our starting position is that technology can make a huge difference to the effectiveness of our work; I just want to see somebody explain to me exactly how this technology will make the specific difference that is claimed for it. So far I haven’t seen that explanation.
Paul Currion
31 Mar 09 at 17:10
Thanks stimulating the conversation, Paul. Just got into Khartoum and will try to reply soon–lots more I want to say.
Patrick Meier
31 Mar 09 at 18:38
[...] 31, 2009 · No Comments This is a response to Paul’s excellent comments on my recent posts entitled “Internews, Ushahidi and Communication in Crisis” and “Ushahidi: [...]
Crowdsourcing in Crisis: A More Critical Reflection « iRevolution
31 Mar 09 at 21:23
Hi Paul, thanks again for your comments on my blog posts. Here are my thoughts in response:
Crowdsourcing in Crisis: A More Critical Reflection
http://irevolution.wordpress.com/2009/03/31/crowdsourcing-in-crisis-a-more-critical-reflection/
Looking forward to continuing the conversation!
Thanks again
Patrick Meier
31 Mar 09 at 21:27
@Tom Good idea. My significant challenge is that I am limited in what I can write about this publicly much as I would like to, because to more clearly delineate what works and does not at this juncture is to give needless ammunition to those intent on shutting down critical information generation and dissent.
Catch 22?
I have in the past critiqued toolkits, for e.g. Tactical Technology Collective’s Citizen Journalism Toolkit (http://ict4peace.wordpress.com/2008/05/20/tactical-technology-collective-citizen-journalism-toolkit-how-not-to-do-a-toolkit/) and five years ago, wrote in detail about our experience with ICTs in tsunami response (After the deluge : InfoShare’s Response to the Tsunami – http://sanjanah.googlepages.com/thoughtsonictandpeacebuilding).
Today, my work as you know involved pissing off a lot of parties. New media, ICTs and all that jazz helps, but it’s not just boys with toys. These are, as you know, real lives, real issues where real compromises are needed with very unpleasant people to secure larger commitments towards human rights and humanitarian aid. These are never easy challenges, no prescriptive model works. This is why I agree with Paul – we can do a song and dance about new tech, but my own work is informed by a sober appreciation for the unmatched power of a repressive regime to harm, hate and kill.
Take care,
Sanjana
Sanjana Hattotuwa
1 Apr 09 at 2:21
@Paul – “What bothers me is whether a) “crowdsourcing” as a methodology yields more information benefits than costs and b) how useful this information actually is in the form(s) that it will be collected.”
This is a shared bother. From Ushahidi’s own Swift River concept (http://blog.ushahidi.com/index.php/2009/02/04/crisis-info-crowdsourcing-the-filter/) to the ICT4Peace Foundation Crisis Information Management Demonstrator (http://www.ict4peace.org/view_blog_posts-1-v-168.html) that seeks to augment the platform with more robust information qualification and analysis tools (and for a closed, trusted network – not public writ large) there’s work afoot to make sense of the noise of crowdsourcing.
I like the term though, but that’s because I live (and have grown up in) a country hugely partial to violent censorship. More eyeballs on events and processes is for me refreshing, even if they are by definition partial accounts. The danger in crowdsourcing though is that they may not be seen as such, and this is where a single SMS can exacerbate violence hundreds of miles away – rapid onset disasters facilitated by new media!
As for your second points – it depends? A “camjo” with a mobile phone who captures footage of police abuse can post this video to Witness to create a storm of advocacy against police brutality. Quieter initiatives such as oral histories through audio and video, or even celebrating champions within Govt who stand up against corruption, or document, through the eyes of children armed with mobiles, life in conflict or with HIV / AIDS – these can be moving, powerful narratives that lead, over time, to social change. Eyeballs on Darfur through Google Earth catalyses and sustains limited and transient global interest in claims of genocide. An initiative like Wikileaks can support whistleblowers and strengthen transparency and accountability in polity and civil society, including in NGOs.
My gut instinct, forged through the bloody difficult work I do is that tech helps, but I would be the last to suggest that it is a panacea. The metrics of measuring impact, or put another way, building up the evidence base of ICTs actually “making the specific difference that is claimed for it” would be an interesting study for luminaries in the field like you, Tom and Paul to take on?
Take care,
Sanjana
Sanjana Hattotuwa
1 Apr 09 at 2:34
[...] Please join the discussion here. [...]
Cutting through crowdsourcing « ICT for Peacebuilding (ICT4Peace)
1 Apr 09 at 4:58
@Paul – re: Toni Comer
Exactly: it was only when Comer was shown the CCTV clip that she went public and filed a complaint! Instant infamy.
I think this case shows that we have to continue aspiring to have institutions tasked with verifying information and creating lawful consequences for breaches of the law. What happens when we stop believing this is ever possible, and that the crowdsourced info is an end in itself? What would Wikipedia be without a defamation law?
@Sanjana — “The metrics of measuring impact, or put another way, building up the evidence base of ICTs actually “making the specific difference that is claimed for it” would be an interesting study for luminaries in the field like you, Tom and Paul to take on?”
Now there’s a research proposal. I know that John Lannon at the Human Rights Impact Resource Centre has pulled together a shed-load of background on appraisal, evaluation and impact assessment from the informatics world, but I’m not sure what he intends the final output to be. A set of “rules of thumb” for tech projects in human rights, aimed at practitioners would be handy.
I see your Catch 22: should we air the dirty laundry outside, when the haters might burn our clothes? We can start by documenting in good time, and more extensively the “safer” uses of new tech in related areas, and supplement it with discussion amongst ourselves about what these may tell us about more controversial uses. I guess that’s what we doing here with crowdsourcing. We could also tone sh*t down. It’s easy to become very partisan about tech, particularly when we are all competing to find funds to keep the lights on. Doesn’t a gradual over-egging of the capabilities of our favoured tool or approache creep in? It’s quite hard in such a wearying atmosphere to keep comment and critique productive and in good faith – it’s even harder to accept it as such. I think this causes a breakdown of trust, wearing down the will to ask for help and comment privately.
Tom Longley
1 Apr 09 at 10:01
[...] furious from all angles. It is reminiscent of many aspects of the ongoing discussion between Paul Currion and Patrick Meier, about the issues around crowdsourced crisis data. Paul writes that, [...]
G20 update; crowdsourced crisis information live «
1 Apr 09 at 16:01
Hi All, I’ve posted a very relevant post on “Peer Producing Human Rights:
http://irevolution.wordpress.com/2009/04/01/peer-producing-human-rights/
@Paul, you may want to delve into the field of digital activism to find examples of how technology makes the specific difference that is claimed for it:
http://www.digiactive.org
Patrick Meier
3 Apr 09 at 10:44
[...] responded: Is there any evidence at all that the US Embassy’s Twitter feed had any impact at all on the [...]
The Antisocial Humanitarian deals with rumours at humanitarian.info
6 Apr 09 at 9:25
[...] you’ve been reading this blog recently, you’ll know that I am singularly unconvinced by the utility of crowdsourcing in emergencies (at least, the sort of emergencies that I’ve [...]
Ushahidi: crowdsourcing in their own words at humanitarian.info
9 Apr 09 at 16:34
[...] Swift River concept tackles this problem, that Paul Currion has succintly and accurately expressed here. [...]
The pros and cons of crowdsourcing election monitoring « ICT for Peacebuilding (ICT4Peace)
7 Jun 09 at 18:18
[...] in Haiti have concerns about the role of crowdsourcing in humanitarian response, which is something I’ve written about previously. If you don’t want to read that whole blog post, the short version is this: I don’t [...]
On Crowdsourcing, with a big sigh at humanitarian.info
30 Jul 10 at 12:18