Arts & Culture Finance Fair Highlights – 3. Looking at impact
Our final highlight from our Arts & Culture Finance Fair looks at impact, both social and artistic. There’s a whole host of relevant topics here, but for the event we decided to run sessions on the evaluation of artistic output and development of theories of change – topics that we feel are of most immediate relevance to the organisations we come into contact with.
Quality Metrics
Developing a consistent, inclusive approach to evaluating artistic impact is a task fraught with difficulty. Firstly, how do you agree a set of evaluation criteria that can be applied across different organisations, let alone art forms? Secondly, how do you manage the inherent subjectivity of an individual response to a work of art? And this is leaving aside the question of whether an evaluation methodology spanning different art forms is conceptually meaningful.
One approach that’s gathering momentum is the Quality Metrics system, currently being piloted by Arts Council England across 150 of its National Portfolio Organisations. The Quality Metrics were developed through conversations with many different arts practitioners, across different art forms from the ground up, and are broad enough, yet at the same time sufficiently meaningful in order to be able to capture and analyse a measure of impact for different art forms. They include measures like distinctiveness, captivation, provocation and rigour, amongst others.
During the afternoon session of our Finance Fair, John Knell of Culture Counts, a data collection platform that captures impact indicators via the Quality Metrics system talked about its progress to date. Its aim, he said, is to inform funders of perceptions of success of various outputs of arts organisations – triangulating opinions of self, peers and public over a set of indicators, so with a large enough sample size and the right balance of these difference stakeholders, a meaningful, more consensus-driven view of a given output can be developed. Culture Counts is designed to help organisations collate and respond to feedback themselves, in the first instance, and communicate the impact of their output to funders and other key stakeholders. The system is still in development, and John responded to audience concerns about league tables and conditional funding by citing the meta-tag system which means there will be limited pools for direct comparison (and organisations will be in control of their comparison pool as they choose their own meta-tags); and reversing the question around funding to demonstrate how the system could be used to confer more accountability on funders, and shed light on funding decisions.
Theories of change
What is the foundation of an organisation’s social impact? We’d argue that it’s a clearly defined Theory of Change, which explains why the activities it does produce the outcomes it intends to achieve.
In the last session of the day, Seva, the Arts Impact Fund Manager and Greg Winfield, a Service Design Officer from Waltham Forest Council, talked the audience through how to go about constructing a Theory of Change. There was equal focus on the ‘change’ and the ‘theory’ – whilst part of what we help organisations do is stake a distinct claim on the positive changes they bring about in the lives of individuals by establishing causality, it is important to note that this is not an exact science, and impact optimisation requires a dynamic attitude and constant vigilance. In addition, whilst talking through the standards of evidence, the audience stimulated a fascinating and healthy debate about how we ensure that we are interpreting these in ways relevant, meaningful and applicable to the interventions for which we’re hoping to establish efficacy. For example, what would a randomised control trial look like for an educational intervention, and how do we feel about the ethics of establishing a control group? Can randomisation really work without a placebo? We welcome thoughts on this; particularly around inventive ways to establish or support claims of causality without losing or discounting real social value by looking through the wrong lens/trying to put a saddle on a cat.
We would like to thank everyone who contributed so graciously and generously to the event, and thank the entire audience for their time, attention and thoughtful questions. This is a tremendously exciting project to work on, and this is entirely down to the invention and enthusiasm the arts and culture sector is giving to the opportunity the Arts Impact Fund presents.