Dealing with subjectivity when measuring social impact
Impact Measurement and Management is done by people. That means personal experiences and biases inevitably have an influence on assessments. How do investors for impact deal with subjectivity?
Our third edition of the Burning Topics series on Impact Measurement and Management offers insights from Octavie Baculard, President of French social impact organisation Kimso.
Social and environmental impact is closely linked to our values: just think of how the definition of impact itself entails a subjective element. Those involved in the Impact Measurement and Management process are individuals, and personal experiences and biases affect their assessments, even if decisions are informed by data and exhaustive analysis.
When it comes to measuring and managing impact, dealing with subjectivity adds a layer of complexity. Different groups of stakeholders, whether they are affected by or just involved in a social or environmental issue, will have different – subjective – views on it. Investors for impact must work to ensure that impact metrics do not oversimplify complex realities.
How are investors for impact setting the example?
Investors for impact acknowledge the subjective influence on decision-making processes and impact valuation; they adopt different risk-mitigation strategies to manage it.
For instance, they aim to engage a diverse group of stakeholders in the impact measurement and management process so that it will be less subject to personal bias. They also ensure democratic decision-making within the organisation’s team and make efforts to reach internal consensus. They engage in participatory processes when selecting new investments. In addition, they build internal sectoral knowledge to better inform decisions, and often seek support from external experts and/or academics.
Finally, as a certain degree of subjectivity is unavoidable, the most important way of dealing with the associated risks is to be transparent about it. Transparency in this context not only means sharing results, but also the metrics used, the assumptions made, the trade-offs identified and the decision-making processes developed.
*
Measuring What Really Matters – Octavie Baculard, Kimso
As president and co-founder of Kimso, a consultancy specialized in social impact assessment, Octavie Baculard has applied her expertise on the themes of health, employment and social entrepreneurship to the evaluation of about a hundred projects.
When assessing the impact of a social project, our first question is: what does the project change, and for whom? This is where our work starts.
We first define impact hypotheses, for instance outcomes in terms of knowledge or skills acquisition, thoughts, emotions, or empowerment. We will then confront these hypotheses by listening to the people involved on the ground. To “report on what really matters”, we often use qualitative methods. We use tools adapted to the people we interview to make sure we capture their thoughts, such as interviews or focus groups. We have successfully tested this approach with particularly complex or vulnerable audiences, such as people in prison, young dropouts, or people with intellectual disabilities.
At Kimso we are convinced that what is not quantifiable is often the most important aspect of a project. Our qualitative reporting methods allow us to really understand what is at stake, before measuring impact with a quantitative approach. We for instance recently assessed a non-formal pilot education programme, by interviewing women and young Yezidi in camps in Iraqi Kurdistan. The study showed that people felt less lonely and developed their interpersonal skills thanks to the programme, and also highlighted the risks of stopping such a programme. The qualitative method is also the only one that can bring out unintended effects, which are difficult to detect in a quantitative study.
When there is a need for metrics and data, we use quantitative methods. Nevertheless, we always conduct at least a few exploratory interviews. This allows us to consolidate the outcome hypotheses and the subjective indicators that will account for such hypotheses. This work is key to draft a relevant questionnaire that will really account for the outcomes targeted. For example, objective indicators such as the labour market inclusion rate, or the number of people who found a home, can be insufficient, or sometimes inappropriate, to show the impact a programme has achieved. Conversely, indicators such as the number of people who feel ready to look for a job, the number of young people with disabilities who are confident in their ability to succeed in school, or the number of people who feel that their view of homeless people has changed, can illustrate what a project was able to change for the better in a much more effective manner.
In short, if we had one piece of advice for impact investors, it would be: make sure that you have developed the right subjective indicators, which can genuinely demonstrate how a particular project was able to achieve social change.
*
This post first appeared as a LinkedIn article.
