Charities to focus on performance or attribution?

Evaluating your organization’s impact

Understanding how assessing a charity’s performance can help to bring about real change in people’s lives

There’s a battle raging in the impact measurement landscape.

No single organization creates social impact in isolation. It requires all the necessary ingredients coming together in a certain ratio. Photograph: A.Y. Photography/Getty Images/Flickr RF

Our standards of evidence are too low, say experts. To identify the impact of an intervention—in other words to attribute impact—we need sophisticated evaluations that must involve control groups, and preferably randomization.

Our standards of evidence are too high, complain others. It’s unreasonable to expect cash-strapped charities and busy social enterprises to incorporate high-end evaluation into their work.

We should focus on managing performance and give up on unrealistic expectations of attribution.

Of course these are two ends of a spectrum, and there’s no one right answer. But while I am an ardent supporter of evaluation, my hunch is that most of the time we should be focusing on the practical rather than theoretical, and evaluation that helps us manage performance rather than attributing change precisely.

That’s not because I want social purpose organizations to be let off the hook, to get on with delivering impact rather than measuring it. No — it’s because I believe that attribution can almost never be established with certainty, yet is commonly asserted or implied when charities make claims about their results. I don’t believe attribution or proof should be the primary purpose of evaluation and impact measurement—rather it should be learning and improvement.

I started to gravitate towards this view when I was researching evaluation in the field of campaigning. In these cases, change happens as a result of complex, interacting actions by multiple actors, mediated by social, political, economic and technological factors. It is, in general, impossible to isolate a particular element of the system and study its impact. As a result, evaluators in the field of campaigning tend to talk about contribution, not attribution.

Back in the world of charities delivering services, I started to question whether this world was really so different from the campaigning field. Don’t multiple actors, actions and conditions also exist for individuals receiving services?

For example, if a young boy turns away from a life trajectory dominated by gangs, to focus on doing well in school, can we really know exactly which actions (or actors) led to that impact? Was it the teacher who offered a listening ear, and persistent encouragement for the boy to recognise his own skills? Was it the mentor who shared his own experience of making the same transition? Was it the shock of seeing a friend seriously injured in gang violence? Or all of these factors? Or none of them—the boy’s own grit and resilience eventually winning through?

My point is that life—people, families, communities—is messy. Even when it looks simple, the chances are that isn’t once you scratch the surface.

Ultimately, I believe that no single organization, or programme, creates any social impact in complete isolation. So any attempt to seek attribution, I believe, should start from a desire to understand and improve—rather than prove—and must be tempered with a realistic understanding of that intervention’s context.

There are ways that charities can start to explore this without breaking the bank—the Justice Data Lab will compare the outcomes of a given group of ex-offenders with a matched group of a similar profile, for example. Or charities can create a control group by selecting participants from their waiting list. But I would still caution those charities from using such studies to claim that their results occur purely because of their work.

If we want really to understand how change happens, then we need to embrace its complexity. Let’s not seek to isolate and attribute impact, but instead to understand our own contribution, and role, within the system.

Can that be done? My experience of the campaigning field tells me that it can, and that seeking to understand social change from a systems perspective will drive more collaborative, collective attempts actually to deliver it.

If I find, for example, that my mentoring programme’s results are heavily influenced by the behaviour and practices of the participants’ teachers, I start to think I should be directly working with them to help shape a partnership programme.

Working collaboratively requires a leap of faith. Charities’ boards may need to embrace working with competitors, in what is a fiercely competitive sector. Funders need to encourage collaboration, and seek evidence of the impact of coalitions and groups, not specific projects in isolation.

The leaders of social enterprises and charities need to stop obsessing about proving their impact, and instead focus on improving it.

Ultimately, if we’re trying to bring about change within a system, doesn’t it make sense to seek out this rich, complex, messy understanding of the world and stop pretending everything’s simple, and can be reproduced under experimental conditions? If we don’t, we may end up lying to our funders, lying to our beneficiaries, and lying to ourselves.

Tris Lumley is the head of development at NPC.

This content is brought to you by Guardian Professional. To join the voluntary sector network, click here.