Learning about Culture: the insider’s guide to setting up a large-scale evaluation

Becky Swain, Head of Learning and Participation at Arvon, is leading The Craft of Writing, one of five arts and cultural education projects that will be part of the RSA and the Education Endowment Foundation’s Learning About Culture programme. Here, Becky reflects on the opportunities and challenges presented by the process of setting up the project’s evaluation.

About the project

As one of five projects being evaluated through the Learning About Culture programme, The Craft of Writing has a key research question at its heart: how does the opportunity for teachers to work with professional writers improve outcomes in writing and attitudes to writing for the children they teach?

Working with teachers from 96 schools across the North of England, the project aims to reach almost 3,000 primary age pupils. This work continues Arvon’s collaboration with The Open University and The University of Exeter, building on their Teachers as Writers (TAW) research that showed powerful effects on teachers’ confidence as writers and children’s engagement and motivation to write.

This project aims to strengthen teachers’ idea of themselves as writers and to investigate how they can use that to inform the creative teaching of writing.  Teachers will attend two, three-day residentials, and a series of professional development days delivered by Arvon between June 2018 and July 2019. The project is underpinned by the techniques and approaches of professional writers and data from previous experience in delivering TAW. This knowledge is used to support teachers to integrate what they learn from the Arvon experience into their teaching of writing.

Lost in translation

Recently, all partners in the project were invited to a set-up meeting, including a ‘logic model’ workshop, hosted by the Education Endowment Foundation (EEF). The terminology involved in evaluation and research can be challenging. Randomised control trials (RCTs) put words like ‘intervention’ and ‘control’ centre stage, and that can feel somewhat alien from within the arts sector. Not least of these is the idea of a ‘logic model’ or ‘theory of change’.

Arvon had worked previously with a research team to develop a logic model for the TAW project, and found it helpful to piece together an explicit, visual statement of which project activities we think will bring about change for participants involved. We had submitted an initial logic model as part of the application process, which informed the start of our conversations in the workshop:

Click image to enlarge

The logic model workshop included the project partners, colleagues from the EEF and the RSA, and also the team of independent evaluators led by UCL - Institute of Education and the Behavioural Insights Team. It was useful to go through the logic model thought process from scratch together, to ensure that all were given a chance to understand the detail of the project from the outset. It struck me during the process that an effective logic model needs to take into consideration the wisdom of those who understand the nuances of arts learning practice and know best how change happens for the participants involved. One EEF colleague asked, ‘What could break and what’s core?’ encouraging us to focus on key activities within the project that we felt were likely to create most change for participants. The discussion strengthened the logic model and also added helpful information about context and recognising barriers to implementation. This is what we produced together:

Click image to enlarge

What do we want to measure?

Whilst I recognise some of the concerns eloquently expressed in Emily Pringle’s recent blog, I am hopeful that Learning About Culture can recognise a range of methods for assessing and evidencing the changes that arts and cultural projects can bring. From our perspective at Arvon, we are interested both in an RCT that assesses impact on writing and ‘non- cognitive’ outcomes, such as enhanced motivation, confidence and empathy, and in a process evaluation that uses a range of methods.

In our set-up workshop, after agreeing the logic model, we discussed the most appropriate tool to assess pupils writing samples. The current testing regime across schools over-emphasises technical aspects of writing, leaving us with questions about the appropriateness of so-called ‘tried and tested’ and ‘reliable’ measures to assess pupils’ writing. 

We agreed that the Writing Assessment Measure (WAM) - developed to assess children's narrative writing - would be the closest fit for our project, but only once we’d made adjustments  to how different elements of writing are ‘weighted’ in the marking. We agreed that the weighting of the creative/expressive elements in the measure would be doubled.

Looking to the future

I left the set-up meeting with renewed excitement about the project and an open mind. We are aware that some research and evaluation projects within the arts sector seem to be driven by the need to advocate for the value of what we do, to prove and evidence the positive outcomes of the work that we lead. Advocacy is of course key for all of us, but for Arvon, being part of a research partnership has reinforced for us that research needs to be driven by genuine shared curiosity to learn, to ensure that the voice of the teachers and students is at the heart of the research, to test ideas and to be willing to be proved wrong sometimes.

I think there will likely be helpful learning through the process of the trials and we hope for good debate too about research and evaluation for arts learning – foregrounding many forms of research that are valuable and enabling more practitioners in the arts to use many different effective methods in future.

• Know a school that would benefit from the Craft of Writing? Find out more and sign up here.

• Find out more about the Learning about Culture programme.