Last Autumn I carried out an evaluation of a professional development program for aspiring social enterprise leaders in the UK that had been running for a decade. Early in my discussion about the project with the client it became clear that understanding causal links and pathways was going to be crucial to this work.
Some core aims of the evaluation were to understand:
1. Long-term outcomes
We wanted to understand any longer-term outcomes (both positive & negative) experienced by beneficiaries since they left the programme. For some people this meant asking them to reflect back over a period of 7 years so we knew that it would take time and space to allow them to explore their own perceptions of change over this time.
2. The ‘highs & lows’ of the professional development journey
The programme was a 3 year commitment to intense study and action learning which people completed alongside their existing work and personal lives. This presented challenges for many beneficiaries. A straightforward focus on end goals or final outcomes would have failed to fully explore the richness and depth of growth that many beneficiaries’ experienced over their time in the programme. For many beneficiaries the ‘lows’ along the way – those periods of growing self-awareness, self-doubt and low confidence – were essential elements of their overall journey.
3. The full story
We wanted to create enough space to explore people’s experiences without the constraints of a narrow focus on the programmes’ theory of change. The client wanted to develop a very honest understanding of a range of beneficiary journeys in all their glorious ‘greyness’ in order to inform future design and funding of the programme.
Having established the centrality of understanding casual pathways to this piece of work I reached out to the team at Bath SDR and Causal Map for some help! And that help came in the form of StorySurvey.
StorySurvey is an app for gathering causal stories. It is a qualitative, causal survey tool. You simply send a link to a question which prompts people to make connections, for example ‘what factors have influenced your professional development in the last 3 years?’ StorySurvey then asks people for the reasons behind their answers and the reasons for the reasons. Or the other way round; the outcomes of the outcomes. The app combines the information from different respondents into an overview map which helps visualise people’s explanations: where they agree, and where they disagree.
We decided StorySurvey would be a great way to support this evaluation for several reasons:
1. To increase reach
Contacting people remotely I was able to reach more people than through one-to-one interviews. There are other online survey tools out there but, unlike StorySurvey, they would not have been able the capture of casual links and pathways that we were so interested in.
2. To give structure to interviews
I conducted some one-to-one interviews and used StorySurvey during these to guide or support these conversations. This not only helped focus discussions but also saved time when it came to capturing and processing data!
3. The reflective nature of StorySurvey was familiar to participants
As part of the professional development studies the beneficiaries had spent a lot of time honing their self-reflection skills, which meant they felt comfortable completing this kind of exercise.
4. The joy of visualisations!
When I had done the work that I love – the human engagement and exploration through dialogue – I could use the data captured by the tool to slice, dice and present that data. The team at Causal Map supported me with my management of the data and the tools and the end results were fantastic.
The only limitation was the usual one – no one tool is right for every single purpose or audience. I often work in high crisis environments such as prisons, homeless hostels and domestic abuse refuges where simply providing a link to an online survey would not be the right approach. Using the software in interviews however was a great way to address this challenge – and a fantastic experience as a qualitative evaluator who regularly drowns in vast and complex research outputs!
Overall the experience was great and the evaluation much enhanced by the ability to collate and aggregate causal pathway data. From my point of view the exploration and visualisation of pathways and links really helped encourage the client to engage with the data rather than passively consume it. The client certainly valued the visual presentations of the findings and during a sensemaking workshop we were able to use the maps to discuss those findings in a way that broke down the idea that the data itself would provide ‘all the answers.’ I look forward to using StorySurvey again very soon and growing my understanding of its application in different environments.
QualiaInterviews is a web app which does just that. It uses an AI-interviewer to collect stories from your stakeholders. It was designed to find out what people really think through in-depth chat conversations. It’s easy, fast and scalable!
Between September 23th to September 27th, Steve Powell and Fiona Remnant were in Rimini, Italy, to present at the European Evaluation Society Conference. Causal Map participated in many different activities, from facilitating a workshop to doing a demo live. Check out the highlights.
Alexandra Priebe from the World Food Programme, interviewing Ashley Hollister and Sarang Mangi from the DeftEdge on their experience on using QuIP and Causal Map in their evaluation.
