Trusting evidence of impact in the arts

The first of the Cultural Learning Evidence Champions Network webinars looked at how to determine the trustworthiness of claims about the impact of arts activities on educational outcomes. This blog offers a non-comprehensive, but jargon-lite (hopefully!) guide to what you should look for to know if claims about impact are justified and if results are meaningful to your work.

There’s an array of research and evaluation literature out there – from university researchers, independent evaluators or arts institutions and education providers themselves. Once you’ve made it through the technical language, it can be frustratingly hard to know where the truth lies and how much it matters to your own work. This advice should help when reading evaluation reports (that judge the merits of a project/ activity) or research reports (that present new knowledge that’s applicable generally). It’s also a good reference point if you’re thinking about evaluating your own activity.

Why look at other people’s research or evaluation reports?

Whatever your line of work, referring to research/ evaluation from elsewhere is useful in many ways, for example:

• If you want to run an activity you may have to persuade others (and reassure yourself) that it’s a good way to proceed. Knowing what effects similar activities have had on learners in similar contexts might help you make a case for your proposals.
• If you're designing an activity from scratch, it might help to know if anyone else has tried something like it before. If you’re working in the arts, it’s likely that you won’t want to replicate someone else’s work entirely, but there may be parallels with your work that can help make your work as effective as possible.
• If you want to improve an activity that’s already up-and-running, you can get ideas from how others have tried to achieve similar goals, or those who have used similar activity to achieve different goals.
•  If you’re designing your evaluation, it can help to look at how others have evaluated their work. Aligning your outcomes, measures and methods with those that have been used before will provide a benchmark for your success and mean that your evaluation can help contribute more seamlessly to the existing evidence base.

If you’re doing any of those things, however, it’s good to know if your source material is trustworthy...

OK; so, how do I know if I can trust what I read?

The Evidence Champions took advice from Jess Heal at the Behavioural Insights Team, Sam Cairns, at the Cultural Learning Alliance and James Doeser at Culture Case. We’ve combined some of their thoughts, adding ideas from a number of other sources (here, here, and from Alan Bryman’s book Social Research Methods), to summarise what to look out for.

1. TRUTH VALUE – how much truth is there in the findings?

• Key Terms: Credibility – does it ring true? Internal validity – do the tests used to measure a change give an accurate picture of what’s happened?
• What to look for in a report:
  • Appropriate and validated tools: Validated tools are instruments for measuring outcomes (e.g. tests, questionnaires), the accuracy of which has been approved by academics, offering reassurance that they measure what they claim to. However, using the right validated tool is important:  e.g. Children involved in a drama programme designed to improve how they express their ideas in their writing will sit a validated test that measures general improvement in writing (e.g. SATs/ GCSEs). Rather than testing the children twice, researchers could use the results of that test to determine if the programme has made a difference. However, if the test gives as much weight to grammar and spelling as to the strength of ideas, its results might diguise the true difference the drama programme has made. 
  • Prolonged engagement with participants and the contexts in which the research was taking place. For example, in educational research, spending time in schools where the programme is being delivered helps the researcher understand those contexts more deeply and reduces likelihood of overclaiming the significance of any chance results.
  • Triangulation: There are many different sources of data that can help you understand if and how an activity has worked. Sources could be different people (e.g. students and their teachers), and/or they could be different methods (e.g. surveys, tests and observations).  Basing analysis and findings on one data source (e.g. a self-report questionnaire completed by students) is risky. Better if similar results come from multiple data sources - what social scientists call “triangulation”.

2. APPLICABILITY - How applicable are these findings to the real world?

• Key Terms: External validity - can the results of the study tell us something about how anyone would benefit, regardless of context? Transferability: To what extent can results be applied to other situations?
• What to look for in a report:
  • Has the activity been tested in real world contexts (i.e. not in a lab)? Lab tests might not be able to replicate the complexity of delivery in an education setting.
  • “Thick” description: Does the report include an explicit and detailed description of the social and cultural context in which the research has taken place? This can help you to judge whether it could transfer to your context.
  • Random sampling when the sample is large: Choosing at random who participates in the research (the ‘sample’), may make it easier to make general claims about the population as a whole. This works if the sample size is large, but it’s much harder if the sample is small. (this video explains why)
  • Purposive sampling when the sample is small:  If the sample is small, it will be very difficult to make sure it represents the population as a whole. Instead, participants might be selected intentionally, so that a wide range of personal attributes are represented. Sampling this way makes generalised claims harder, but might make it easier to spot if the findings might appply to your work and your participants.

3. CONSISTENCY – Do we get the same results every time with the same/ similar participants?

• Key Terms: Reliability – can these  results be replicated? Dependability – have the proper procedures been followed?
• What to look for in a report:
  • Have the results been based on a single test, or has the test been repeated? Whether it’s with the same participants, or other, similar participants, repeating the test reduces the likelihood of a freak result.
  • If the results are based on subjective judgment (i.e. the opinion) of participants themselves, the researcher, or another observer like a teacher, are those judgments in alignment?

 4. NEUTRALITY - Do the results come from the data, or the biases of the researcher?

• Key Terms: Objectivity; Confirmability – has the researcher acted in good faith and not allowed personal values or beliefs to shape the research or the findings?
• What to look for in a report:
  • Reflexivity: has the researcher acknowledged how their beliefs, values or preconceptions might have affected the research?
  • Have the raw data been made available? (N.B. This isn’t always possible, especially with large, qualitative data sets.)
  • Is there a description of how data have been collected and analysed?
  • Are the instruments/ measures standardised or are they unique to this study? Standardisation makes it less likely for them to be influenced by the researcher/ programme designer’s values or preconceptions.

 Inevitably, these brief descriptions can give rise to more questions – please post them below and we’ll do our best to answer them. We’ll also continue to blog about the Evidence Champions Network and its conversations, so watch this space for more!