In recent years, with voluntary and community sector organisations (VCS) increasingly responsible for delivery of public services, requirements and expectations around measurement have grown. This has added to the pressure already experienced by those working in speech, language and communication (SLC) following the Bercow Review in 2008. The research commissioned for this review highlighted the need for good strategic planning at a local level for children and young people’s SLC services and pointed to a lack of effective use of good quality data to inform policy making and commissioning. It stressed the importance of identifying best practice and the need for evidence of effective interventions for children and young people with different types of SLC need.
However, nearly ten years on from the Bercow report, many small and medium SLC organisations – like others across the voluntary and community sector - are struggling to evidence their effectiveness. They are often unable to provide their commissioners with good quality data about the number and type of people they support, articulate the rationale for their approach or provide good quality evidence for the changes that result from their intervention.
So why are small VCS organisations finding measurement so difficult?
Many small VCS organisations feel under pressure to adopt a ‘good science’ approach to assessing whether or not they are doing a good job. Our experience of supporting VCS organisations across a range of areas suggests that this focus may be hindering rather than helping them to understand and measure their effectiveness.
The ‘good science’ approach to evidence often encourages leadership teams to focus on finding ways to gather experimental evidence that their ‘unique’ approach ‘works’. This ignores the reality of the context in which these organisations are operating - that successful approaches usually ‘work’ because they are highly individualised/responsive to the specific needs of the beneficiary rather than because they follow a set intervention model. Charitable services can be particularly complex social interventions often addressing multiple needs and involving multiple agencies. These types of interventions rarely fit the ‘good science’ approach leaving many VCS organisations not knowing where to start when it comes to gathering evidence or pursuing expensive research designs that result in poor quality data.
The ‘good science’ approach to evidence discourages good strategic planning
A key issue is that the ‘good science’ approach encourages a black and white view of effectiveness - it either works or it doesn’t – rather than a view of effectiveness as an infinite scale for improvement. In this context, questions that are key to good governance (and understanding and sharing good practice) can get overlooked. By concentrating on demonstrating that their intervention works, organisations often fail to ask themselves what they know about the needs of their beneficiaries, what they have learnt from their stakeholders about their effectiveness in meeting those needs, what changes they are going to make to improve their services as a result and what learnings they can share externally about the keys to success in improving lives of disadvantaged people.
Focusing on ‘good governance’ rather than ‘good science’ will improve the quality of evidence
Service delivery organisations rarely have the in house research expertise, support high enough numbers of people, or can claim to be independent from the intervention they are evaluating for a ‘good science’ approach to be appropriate. Much better in this context to adopt a ‘good governance’ approach – one that gives equal weight to measuring effectiveness (using proportionate methods concentrating on short term outcomes) with learning how to improve the efficiency and effectiveness of services. A ‘good governance’ approach to evidence will improve the quality of data collected – not only enabling VCS organisations to improve their services in the future – but ultimately enabling them to share learning and good practice and make a significant contribution to the body of evidence for the whole sector.
Anna Grey, Evaluation Consultant and Co Founder of Cornish and Grey
Cornish and Grey