Exploring evaluation complexities

I’m in the middle of evaluating the Arts and Social Change strand of Citizen Power Peterborough. I don’t want to get into the details of the programme itself – read here if you'd like a primer – but rather, to talk about a few interesting problems that the evaluation has thrown up.

Evaluating something like Arts and Social Change isn’t about measuring ‘success vs. failure’ – if everything in the project was a ‘success’ in that narrow definition, then there would be no learning and the project as a whole would have failed. Citizen Power Peterborough has above all been an experiment – and nowhere more so than in Arts and Social Change. The goal is to find out what impact, if any, the arts can have on positive social change, and this has been pursued through a number of targeted arts-based interventions in Peterborough. Some projects have been hugely successful in terms of impact, others partly so (with important findings), and all have been able to adapt as they progressed, reflecting on-the-ground realities, new ideas and preliminary results.

The Arts and Social Change programme has run according to a set of principles; one of those principles is emergence. To paraphrase broadly, this is the idea that interventions in complex structures (like the communities of Peterborough) will lead to multiple, complex outcomes – the kind that can’t easily be predicted at the outset. These kinds of findings are extremely valuable, because they can only be brought to light through hands-on experimentation.

So, to recap: a huge experiment in a complex structure, where accurate prediction is all but impossible, where there are high levels of reflexivity, and where only the broadest of goals (increasing attachment, participation and innovation) were known at the start. How do you evaluate an experiment like the above?

One tactic is to do what many people would do when faced with a big problem: break it down into a series of smaller, more manageable problems. Arts and Social Change ran as a series of interconnected strands, linking with other parts of the Citizen Power programme: these strands were much smaller and more responsive, with fewer participants from all sides. They had more specific goals (such as ‘increasing community cohesion’) and tentative measures for their individual success or failure. Evaluating the strands themselves in this way will certainly be part of the final evaluation, and it’s incredibly exciting and positive to get to delve into the programme at that kind of level.

It would be missing a trick, though, to evaluate the whole programme by the success of its parts. Talking to people involved, one of the programme’s real (and if we’re not careful, hidden) successes has been its impact upon the ‘bigger picture’. To give an example: one of the first documents I came across whilst researching, was a letter to the Evening Telegraph (Peterborough’s local paper) from a resident, describing an intervention that had been quite strongly criticised by the paper: “…I found it one of the most enlightening and thought-provoking activities that I have ever taken part in. I still find it hard to believe that the city council had the courage to help fund this, but I am very glad that they did.” Read her words carefully once more, and try to recall the last time a Council-funded programme made you feel that way. How do you measure enlightenment? Was it ‘good value for money’? The author measures the cost favourably against some other council spending (and she makes a convincing case), but could you price the “most enlightening and thought-provoking” events in your life? I know I couldn’t. Impacts like this, if they can be nailed down and cogently articulated, give the lie to those who see the arts as an ‘optional extra’ – a luxury to be cut when money’s tight.

Consider this: I like knowing my neighbours, but I have enough social capital that I don’t rely on them – if I have personal or professional difficulties, I have plenty of places to turn to. I like where I live, but if I had to move, I’m pretty certain I’d be fine. It’s not like that for everyone. We’re talking about real interventions in places where community ties, family bonds and professional networks are all under incredible strain, and where without support, a space for dialogue and the ability to explore together, things are unlikely to improve. Art can make that happen, in a way that little else can, and Arts and Social Change is in a unique position to show how. I’ve heard neighbourhood managers talk about how an intervention has fundamentally altered how they see their work, civil society leaders tell of a re-invigorated sense of collective self-belief, and residents describe moving from isolation, to feeling that they are involved in a shared project – a shared life – with those around them.

But how to capture all that? We’re all going to face some extraordinary pressures over the next few months and years, and Peterborough will face as many of them as anywhere. If we can articulate the many things that have been learned by Peterborough’s residents, then we can share them, and play a part in handing powerful tools (for free!) to communities who need them most.