Following your hunches – changing how we think about evidence
Jenny North | Deputy CEO | @JayEeeEnn
This is the text of a speech delivered by Jenny North, our Deputy Director, at the Children & Young People Now Conference on 5th December 2018.
Despite our very modern name, the Dartington Service Design Lab is a long-established research charity dedicated to improving the lives of children and young people by helping the systems and services which support them. We work with charities, local authorities, and funders to help them to strengthen the design, delivery, and monitoring of what they do. Our track record and learning over 50 years, particularly over the last ten, has convinced us we need to change the conversation about evidence.
Something we’ve learned a lot about, and so talk a lot about, is that it’s not possible to get a ‘result’ from evaluation that can make you totally confident about your impact – even a Randomised Controlled Trial doesn’t tell you if it’ll work in the future and usually doesn’t explain exactly what led to a positive impact. We see the effects of this in the fact that often ‘evidence-based’ programmes can’t replicate their impact when delivered in different places and at different times. Local context matters – and the team that deliver matter too.
Whilst any evaluation design – Randomised Controlled Trials included – can’t create certainty, a variety of methods can increase your confidence that you are reaching the people you want to, that your service is engaging them, and helps them to make progress. These can add up to a wealth of knowledge about how well-set up you are to make the difference you want to. We want to support more organisations to assemble this kind of knowledge and then think critically and honestly about where their evidence shows they could be doing better.
This is a different approach to evaluation. At the Lab, we’re never going to go back on our belief that organisations need to spend time and money on data collection and data analysis. But one of the things we feel most strongly about is putting control of data and evidence in the hands of organisations themselves – it’s for your learning and change, not as part of a ‘prove yourself’ agenda, which hasn’t got us very far, and which charities have described to me as feeling ‘punitive’.
Changing the conversation doesn’t mean letting ourselves off the hook and assuming everything’s great. We start from the certainty that your service definitely doesn’t work as well as you’d like, for everyone that you’d like it to. That’s inevitable. What’s not inevitable is not doing anything about it. We’ve all got the opportunity to do something about that.
At Dartington, we try hard not to start our conversations with partners by talking about evaluation methods. We start from helping our partners frame the question they need to answer. This is harder than finding a question that suits a method, but more rewarding. We begin with a review of what’s already known – either from formal evaluation, or from the routine data that an organisation might collect about who the clients are, how they engage with the service, and what it seems to deliver for them. From this data, our partners can identify gaps or problems that are stopping them being as effective as they’d like. The more they know, the more they see things they want to take action on:
Do they want to know how to reach more, or different, people?
Do they need to know how to improve the outcomes they’re achieving so they’re closer to where they want to be?
Sometimes it’s not just about their data. Maybe their staff feel they have a better understanding of their clients’ lives than they used to – should they (our partners) change what they deliver to reflect this?
When our partners have these type of questions, Rapid Cycle Testing can be a useful method. In a nutshell, it looks like this:
Determine where in your delivery you have an opportunity to make a change in the name of being more effective. It could be anywhere (or in multiple places): outreach, enrolment, content, duration, exit. What’s your hunch, or hypothesis, about what might work better?
Follow up that hunch with staff, clients and partners: assemble a co-production team to design an adaptation, or innovation to your service, as precisely as possible, with a clear idea of what it’s trying to do. The question you ask will be around how well this change can work to fill the gap you identified in your effectiveness.
Implement it as well as possible – and this usually involves providing extra support for staff tasked with actually seeing through the change.
Test it – in two parts:
Is delivering this change feasible for your staff and acceptable to your users. It might look great on paper, but does it fall apart on contact with reality?
Is it making the difference you hoped? Is a new outreach strategy reaching different people? Is it changing the structure of the programme increasing attendance? Is aligning the content more with what’s going on in their lives allowing them to engage more and make more progress?
Whatever the question you were trying to answer, Rapid Cycle Testing won’t help you prove it, like a Randomised Controlled Trial – so don’t sweat collecting tons of data. The art of Rapid Cycle Testing lies in choosing the data points, and the quantity needed, that will give confidence that this change is doing what it’s supposed to. You’re looking for enough evidence to make your next decision: does the change become business as usual, do you stop doing it, or tweak it further to test again?
I want to go back to what I said earlier – that Rapid Cycle Testing comes best out of having a bit of knowledge from data collection and analysis, or evaluation. You know some stuff, and some of that stuff will be about what’s not going so well. From this partial picture, you can develop hypotheses – or ‘hunches’ might be a better word – about what might work better. I want to tell you briefly about the three hunches of different organisations we know well – and what’s led them to Rapid Cycle Testing.
1. The first organisation is a small mentoring charity who had commissioned a robust experimental evaluation which revealed a lack of impact compared to the control group. When you looked at the results, and the programme, it wasn’t easy to identify why. The programme was long, the volunteers were carefully selected and well-matched with the mentees, the charity provided lots of supervision – all things that the evidence base says are important.
So, we looked at the young people being served – they were facing significant issues. The staff and volunteers knew this – but the content of the programme wasn’t really addressing the reasons they were referred to the programme or the underlying problems that meant they were struggling.
So, they’re rewriting the content – drawing on evidence of course but also the experience of the staff and volunteers over many years. We’ll be testing the hunch that content more tightly linked to the children’s issues and lives will help them make more impact.
2. The second organisation is a large provider of an evidence-based programme with national reach and many staff. A new experimental impact evaluation showed they weren’t having the effect they wanted in England. They were shown to be delivering the programme with fidelity. But this led to one of the hunches – that actually delivering the same programme to all the clients regardless of their personal context might be undermining impact.
So now we’re helping them to test personalisation. This involves making a better, more precise assessment of the client’s specific needs beyond just their eligibility criteria. Then practitioners can ‘dial-up’ or down the intensity of the service and tailoring the content of what she receives.
3. The third organisation is another large provider – this time with quite impressive evaluation results. They were proven to have an impact on one outcome but not on another that was equally important to them. Their hunch is that they were not retaining clients long enough to have an effect on that outcome, which they suspect takes a longer engagement to achieve.
They need to reduce attrition so that clients stay longer with them – so they are trialling ways to make the later parts of the programme as sticky and engaging as the first.
What all these organisations have in common is that they’ve faced the fact that they can improve. This is a very different starting point from ‘prove we work’. Using data for improvement means asking different questions and using different methods than previously. The organisations above are their own audience for these evaluations. The confidence they are trying to increase is their own. We would like to see many more organisations – as well as their funders and commissioners – asking themselves: What’s your hunch about what’s not working as well as it could? What would you try to change that? And how would you know if it was working?