This is the second of our blog posts from the Making an Impact workshops we ran in August. Sarah Bottle (Service Manager: Innovation, Digital and Libraries) and Darren Smart (Strategic Manager Operations for Libraries, Registration and Archives) talked about Kent’s framework for evidencing impact. The full set of slides can be viewed online, but we’ve summarised the highlights and included answers to some of the questions that were asked during the workshops in this post.
What is it?
Kent have built a tiered framework with select outcomes and tags through which it can quantify stories based around the chosen outcomes.
So how does it work?
1. Collect stories
Staff collect stories from users, whether it’s through active listening or asking open questions to tease out more detail and noting down feedback on one of the A6 forms found at every service point in Kent’s libraries.
When talking to users it’s important that staff ask follow up questions where needed to ensure it’s a good story. Such as the one below:
The form says: a customer was recovering from a complete breakdown and still suffers from anxiety and depression. The mindfulness colouring group has made a real difference in helping her regain social skills she had lost. This form is an example of a good story as it has clear outcomes.
2. So what? panel
Stories are then input into the framework (a spreadsheet) and coded by the ‘So what’ panel against the agreed outcomes. The ones Kent uses in the framework come from the county council’s strategic outcomes and the Libraries Deliver outcomes.
Kent’s strategic outcomes are:
- Children and young people in Kent get the best start in life
- Kent communities feel the benefits of economic growth by being in-work, healthy and enjoying a good quality of life
- Older and vulnerable residents are safe and supported with choices to live independently
The spreadsheet also has a tab which shows summary information.
Stories are saved separately and linked to the entry in the spreadsheet. Kent’s ‘so what’ panel is made up of a data expert, a frontline member of staff and a middle manager. They meet every 2 months and review around 100 stories [Kent have 99 libraries so it’s unlikely most services will get as high a figure]. Sometimes the panel need to follow up with staff to get more information for the story to be a great one with explicit outcomes. When they first started, 20-25% of the stories collected across the county were ‘great’ and now that’s 50-60%. Darren also QAs the panel by spot checking stories to ensure consistency.
Rules for all panels:
- Rule 1: have an odd number of people (so there’s always a majority vote which overrides any seniority on the panel)
- Rule 2: have a good spread of people with different perspectives or views for a balanced panel, all of them must understand the Outcomes focus / nature
- Rule 3: keep the panel the same - but if membership needs to change, rotate them in and out (only replace one at a time and ensure panel diversity is maintained), this helps consistency and ensures objectivity
- Rule 4: don’t over-interpret the stories, they shouldn’t have more than 5 outcomes, on average you’d expect 2 or 3
All stories are anonymous, so no permission is required to share them. Staff names are recorded on the comments sheet though, in case clarification or more details are needed. Sometimes, stories are so good they can be developed into case studies or videos for more impact. If any direct quotes, photos or names are used, it is with permission (staff will be able to identify most individuals again).
The panel also tags stories with certain demographics or activities which it may want to collect data on, for example Kent has a refugee tag.
3. Surrogate measures
Kent have created a database of research which can be used to layer over the stories data to show how the service is making a difference which all creates a powerful picture.
Stories in practice
Through attendance data, Kent can show that 72 Talk Time groups were held in 2016/17 with 29,000 people attending. The spreadsheet shows that 25% of the stories demonstrated that libraries contribute to combating social isolation. Stories are strengthened with videos from users like Roy and are combined with surrogate data. It has been calculated that loneliness increases the likelihood of mortality by 26% in older people - to create a powerful picture of the impact of library activity.
Outcome educating
To set up this new system, Kent started by educating all members of staff about outcomes. They shared examples of stories which were:
- good (as shown above)
- could be great
- not so great
Could be great stories like the one below (You are so efficient in here. Thank you so much, you have been very helpful) are ones which could have been a great story if the right follow up question has been asked. For example: How have staff been helpful? Why are they efficient? What difference did you make? So what? The ‘so what’ question was coined by a member of staff and became the name for the panel which codes the stories.
A not so great story below just says: what a lovely table. How fabulous 🙂 and again leads to the question ‘so what’?
Embedding the process in the council
It took Kent about 5-6 months to embed the process and get staff on board. Kent used a variety of methods to communicate the new process:
- emails to all staff and volunteers which include examples of stories, sharing why it’s important and how the stories are going to be used
- staff in each of Kent’s districts went to a workshop on the new process and they then cascaded to all libraries in their area
- outcomes champions were appointed in each district
- filmed a webinar which provides an overview of the outcomes process and is available to all staff on Kent’s youtube channel (and now features as part of staff inductions)
- performance development forms for people in a supervisory role now include a line on promoting a culture of outcomes (not the number of stories collected)
- managers visited all libraries across the county to talk about the process
Building up an evidence base
It takes time to build up a body of evidence to show impact but one way to increase your stories early on is targeted evidence gathering. This could be through observation, one to one interviews or group chats on specific topics, and is a good way to build up evidence in a short space of time. For example, Kent ran one on the Summer Reading Challenge, using targeted questionnaires, interviews and groups discussions and observation.
How is it used?
Kent is an internally commissioned service which prepares regular reports to the cabinet. The reports are now outcomes focussed (and shared with all staff) who can see the changes from the previous KPIs format. The framework has given them an advocacy tool from which they can pull out a variety of statistics based on requests for information, for use in funding applications and to inform decision making.
If you would like more information on how to do this or would like a copy of the framework please email darren.smart@kent.gov.uk or sarah.bottle@kent.gov.uk.
To read all the posts relating to this workshop, search: #LibrariesImpactMeasurement.