Blog: Is what works what matters?

Matthew outlines some key lessons from our work with education company Pearson on the strengths and potential pitfalls of focussing on efficacy. The first in a series of three blog posts exploring efficacy.

I have written some rather scathing comment on corporate social responsibility recently. But I stand by my enthusiasm for the approach taken by the global education company Pearson. Under the guidance of my old friend Sir Michael Barber, Pearson has a commitment to ensure that all their educational products and services genuinely ‘help people make progress in their lives through learning’. This means a challenging process involving the whole company worldwide and will include terminating services or products, even if they are profitable, that fail a robust efficacy test. Unlike so much CSR, this initiative takes responsibility to the very heart of the company’s business model.

Prompted by this initiative the RSA held a series of seminars with Pearson and other partners to explore dimensions of efficacy. We will publish a final report on that process on our website soon, but I have drawn out some of the main lessons in this post, while tomorrow I will explore and further develop the analytic framework that emerged.       

Efficacy is ethical: Efficacy can feel like a dry technocratic topic. Its pursuit can also feel burdensome. Inertia is, after all, an inevitable part of the human condition; if we constantly questioned the meaning and usefulness of everything we did we would end up immobilised and depressed. But for organisations seeking to generate public value, being willing to ask 'what we are doing, why we are doing it, what we are achieving and whether we are getting better' is crucial.  Without having a commitment to measuring efficacy and to putting in place systems to enable a robust focus on impact, organisations will fail to generate the good that they could.

Efficacy can be democratic: Efficacy is an idea that often drives accountability upwards to organisational leaders. But, as Pearson has found, the discussion about ‘what we are doing, why we are doing it, what we are achieving and whether we are getting better’ can and should be a tool for engaging people at all levels of the organisation and key stakeholders, including customers and the public. People want to work for organisations that are making a difference so debating and acting on efficacy can help motivate as well as engage. 

Efficacy can seem irrelevant: Data analysts, policy wonks and strategists tend to assume that everyone cares about efficacy. But, as Pearson sometimes discovered when they tried to shift people to more effective educational interventions, people have other motivations. Asking hard question about efficacy and calling existing practices into question can be disruptive. Self-interest, path dependency and avoiding conflict can all stand in the way. In short, the advocates of a focus on efficacy should not take the power of their case for granted.

Efficacy is complex: Efficacy can feel like a relatively simple idea – analyse inputs and outputs, come up with an intervention and then seek to improve that measure. The reality is often very different. In even relatively straightforward cases, identifying inputs, defining and measuring outputs and then explicating the link between inputs, outputs and outcomes is hard work: Pearson’s process is detailed and exhaustive. Invariably, in the real world, the pursuit of efficacy is taking place against the background of several other intervening variables. This is one reason why, when it comes to interventions in public services, which involve changing human behaviour, successfully replicating practice that has worked elsewhere has often proved challenging. Indeed, even in the more scientifically controlled environment of medicine, the rate at which ‘proven’ techniques turn out to fail (sometimes called ‘medical reversal’) is surprisingly high.  

Efficacy is contested: A recurrent question in our seminars was ‘efficacy for whom?'. Our efficacy in schools seminar explored whether the OFSTED-enforced use of techniques to improve standards, which arguably drive higher exam performance, is compatible with a rounded education for children, professionalism in teachers and creativity in schools. Similarly our seminar on employment questioned the Government’s conviction that stronger conditionality and sanctioning are efficacious. They may be so if the goal is that people take any job on offer however lowly paid, poorly skilled and short term, but what if we see the dignity of claimants or the importance of work providing a decent chance of progression as important criteria for success? Even within organisations – as anyone in the NHS could tell you – what counts for efficacy at the top may emerge at the bottom as practices which are pointless or even counter-productive. 

The well-meaning pursuit of efficacy can sometimes make things worse: Efficacy initiatives are prone to suffer from Goodhart’s law, namely ‘when a measure becomes a target, it ceases to be a good measure’. The classic example was when university academics started to be rated (and, indirectly, paid) by the number of articles they placed in refereed journals. This may initially have been a good measure of how productive academics were, but the consequences of making the performance the target included the spread of new journals of questionable value, the tendency for academics to dress up the same research in different ways for different journals, and the naked attempt by universities to sign up published academics in order to raise their research ranking. Partly to address this, the Government is now planning a teaching measure to set alongside the research measure, but judging by the weak proxy measures for teaching quality proposed by ministers (student satisfaction surveys, job destinations and dropout rates) there is plenty of scope for more gaming.

The first conclusion we can draw from our seminars with Pearson is that efficacy is a vital tool but it must be handled with care. One way we might become better at using that tool is to recognise that in fact there are three different ways of thinking about efficacy, but that is the subject of tomorrow’s post.  

• Read Matthew's second blog post on efficacy: Three, two, one - impact! 
• Read the third post on efficacy: Departments, collaboratives and projects - how organisations maximise their impact