Guest blog: The Reading Outcomes Framework Toolkit – an evaluation partnership

There is a growing expectation for arts and cultural organisations to measure the impact of their work but developing an evaluation framework can be a challenge. Seeing the need for sharing examples of best practice and know-how, we talked to The Reading Agency, which recently published a Reading Outcomes Framework Toolkit. While not all organisations will have the resources for such an in-depth investigation, we think their example highlights key principles that others can incorporate in their impact management:

• proactively trying to understand the full spectrum of outcomes an activity leads to
• using a collaborative approach to validate results and draw on additional resources
• using findings to improve future practice and inform other similar interventions, helping to make outcomes measurement comparable across sector organisations

The Reading Outcomes Framework Toolkit – an evaluation partnership

by Sue Wilkinson, Chief Eexecutive of The Reading Agency. Follow Sue on Twitter @sue_wilkinson1

Osborne and Gaebler (1992)* said:

• What gets measured gets done
• If you don’t measure results, you can’t tell success from failure
• If you can’t see success, you can’t learn from it
• If you can’t see success you can’t reward it.

These principles have very much driven our approach to evaluation at The Reading Agency (TRA). We think that evaluating the impact of our work is critically important in helping us to achieve our ambition to inspire more people to read more. We know that we can do this more effectively if we are clear about the difference reading for pleasure and empowerment has on people’s lives and can use learning about what works well to develop high quality experiences that can increase people’s enjoyment of and engagement with reading. We believe that everything changes when we read but to be taken seriously we need to prove it.

In 2014, we bid to the Peter Sowerby Foundation for funding to help us identify the outcomes of reading for pleasure and to develop a set of tools to help us to measure these so that we can learn more about the impact of our work. As a result of their generous investment, we have been leading an innovative, collaborative project to develop a Reading Outcomes Framework Toolkit to help improve impact evaluation not just in our own organisation but across the sector.

Throughout, we have been working in collaboration with  other stakeholder organisations: Arts Council England (ACE), Association of Senior Children’s and Education Librarians (ASCEL), BookTrust, Chartered Institute of Library and Information Professionals (CILIP), National Literacy Trust (NLT), Publishers Association (PA), Scottish Library and Information Council (SLIC) and the Society of Chief Librarians (SCL). Working with a wide range of partners, our goal was to identify evaluation tools and expertise that could be shared across organisations, enhancing our ability to measure impact, drive change and make the case for investment.

We have worked with our stakeholders in a range of different ways:

• Our core Steering Group, which includes the partners listed above, meets quarterly to oversee the development of the Reading Outcomes Framework Toolkit and ensure collective ownership;  
• A smaller Working Group of research experts leads and delivers particular activities, for instance turning consultation findings into a workable product;
• A larger virtual Advisory Group  made up of stakeholders from across a range of reading related sectors including academics and other experts from within both the reading and evaluation fields has provided critical friends supporting the development and delivery of project.

This partnership approach has been our key to success. It has enabled us to draw on an impressive range of research and evaluation experience from across the sector alongside detailed knowledge of what is needed, wanted, will work and will be used. The work has been collaboratively owned and developed, with partners actively involved every step of the way; helping to source research, identify evidence, debate approaches and model the framework.  Early discussions at Steering Group meetings, for example, included defining exactly what is meant by ‘reading for pleasure’: When you feel sad or scared or angry after reading a book is this still ‘reading for pleasure’? When reading is hard can it still be fun? Is the enjoyment of reading fundamental to the activity?

How we developed the Framework

The first stage of the work was to commission a literature review from BOP Consulting to help us to identify and agree the outcomes we are trying to achieve through the reading programmes we deliver.

We then ran a series of workshops across the UK to finalise those outcomes and identify robust tools to help us measure them. As part of this process, we consulted with a range of people engaged in promoting reading including from libraries, the health sector, teaching and the third sector.

In October 2016, the Reading Outcomes Framework Toolkit was launched. It includes:

• A framework of the outcomes of reading for pleasure and empowerment;
• Sample survey questions to evaluate whether a programme impacts on these outcomes;
• A bank of evidence about how reading relates to these outcomes.

The Toolkit is freely available and can be used by anyone working to encourage reading for pleasure and empowerment, for example third sector organisations, libraries, schools, colleges, early years settings, prisons, family learning professionals and the health sector.

We hope that colleagues will use it to help them to understand, demonstrate and improve the impact of activity aimed at encouraging reading. In addition, it can support you in making the case for investment through the evidence it helps you to collect as well as through the existing evidence bank.  

This year, as well as showcasing the framework and the collaborative process which has created it, we have also had investment from Arts Council England for five organisations to pilot the use of the toolkit to evaluate programmes they are running. The resulting report, due to be completed at the end of 2017, will both articulate the findings and feed into the further improvement of the Framework and tools. In addition, Book Trust has invested in the next stage of the work by leading a review of the measures of reading engagement outcomes to enhance the resource base supporting the Framework.

There has been enormous interest in this work from a wide range of organisations in the UK and Europe from across the library, education, health and charity sectors. We have shared the learning at a number of events recently, and held training sessions with smaller groups. We will continue to do so in the future. The feedback so far clearly confirms the original vision that Peter Sowerby so generously supported that a shared evidence based approach to evaluating the impact of reading would not only help us to better articulate the difference that reading makes to people’s lives but also to improve the work we do to deliver it.

Get involved

The reading outcomes framework toolkit is an interactive pdf available here. It is best accessed using Adobe Acrobat Reader on a PC. A print-friendly version of the toolkit is available here. The outcomes framework is available separately here.

If you use the toolkit please send comments to: [email protected]

The Reading Agency and its partners would like to know what you think about the Toolkit, whether you’ve used it or not, so they can inform its future development. You can share your feedback with them by 8 September 2017 using this link: https://www.surveymonkey.co.uk/r/RLFPX6G.

*Osborne, D. and Gaebler, T. (1992) Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector. Addison-Wesley Publishing Company Inc., Reading