It is hard for organizations that address complex social, economic, and environmental challenges to determine what strategies may have the greatest impact. This should be easier with more organizations publishing what they learn and AI tools supporting efficient curation of insights. In reality, the vast amount of data generated is met with overwhelm, and impact-focused organizations struggle to determine what data can be meaningfully used to support their strategic ambitions and what is just noise.
Over the past ten years of working with foundations, government agencies, and private sector organizations, Caribou has found that impact evidence synthesis is one of the more powerful tools to support strategic decisions. Evidence synthesis is the process of compiling and analyzing multiple studies to provide comprehensive impact insights on a set of interventions or products.
In this article, we share our experience of the value that evidence synthesis has catalyzed, the need to improve prevailing impact evidence synthesis norms, and how we have refined our evidence synthesis processes and products to drive impactful decisions.
The power of leveraging collective insights
Organizations that address complex social, economic, and environmental challenges are not doing it alone. Scattered across the internet and within these organizations’ own knowledge management systems, there is an abundance of insights that could be harnessed to explore the pertinent questions of what works, under what conditions, and for whom. Impact evidence synthesis brings these insights together.
Individual research efforts provide puzzle pieces that evidence synthesis assembles to create the impact picture. It adds immense value by:
- Generating a holistic view: Combining varied perspectives provides a comprehensive view of an issue, allowing for inclusive assessments of what works and what is untested.
- Optimizing resources: Identifying promising interventions and avoiding duplicating those with limited impact are valuable tactics in light of finite resources.
- Enabling adaptive strategies: Creating an information infrastructure supports agile strategic responses to the changing social and economic landscape.
By basing actions and investment decisions on a curated body of evidence, organizations can establish a clear and justified basis while also building an evidence ecosystem and infrastructure that encourages a collective approach to engaging in shared issues.
Challenges within the prevailing forms of impact evidence synthesis
There are two phases to unlocking the value of evidence synthesis. The first is intentionality on what is included in a given synthesis, and the second is ensuring the outputs are delivered in ways that support access and — fundamentally — the use of evidence. While there are diverse views on these phases, there are dominant practices. These dominant practices serve some sectors well; in others, they present challenges that can limit their value. Two of the main challenges are:
- A bias toward experimental methods: In some sectors, an exclusive focus on one methodology can be challenging for numerous reasons, including issues of power, diversity, and representation, and discounting early insights in emerging and rapidly changing sectors — like digital — where experimental evidence may not be available when investment decisions are being made.
- Static representation: Evidence synthesis products often exist as static narratives that summarize the state of evidence. The last decade has seen an increase in evidence maps: graphic representations of the landscape of impact evidence plotted against a set of interventions and outcomes. Maps vary but are usually one-off and focus on visualizing the number of studies rather than the insights gained from them. Unless they are accompanied by narrative summaries, evidence maps leave impact insights on the table. Like all knowledge products, content and medium directly affect engagement and use by different audiences.
At Caribou, we have struggled with these prevailing norms and forms, which sparked our work to innovate and improve the practice of evidence synthesis.
How we’ve refined our approach to connect insights to decisions
Our approach seeks to provide timely insights with rigor, context, and diversity of perspectives. Five elements distinguish our evidence synthesis approach:
- Iterative Theory of Change: Designed with thematic experts, the Theory of Change outlines impact pathways for specific issues as the basis of the synthesis.
- Diverse and credible impact methodologies: Experimental and theoretical approaches each offer important views on the evidence. We need both and include both.
- Impact results: We include a view of the volume of data points that initiated positive, negative, or no impact, not just the number of studies on a given topic.
- Design and delivery mechanisms of interventions: Interventions or products (e.g., upskilling, e-commerce platforms) are not homogeneous. They have unique design and delivery features that can and do affect their impact. We include these features in our analysis to really drill down into what works.
- Contextual insights: Who is being studied and where influences impact. We extract contextual data, leveraging localized evidence to support targeted interventions that are more likely to succeed.
Including and extracting this level of detail from individual studies supports a rapid and deeper interrogation of the impact data — many of these elements are visualized in our existing evidence maps on digital financial services and digital and data-first approaches to driving small business growth.
Transforming the way users interact with evidence synthesis
At Caribou, we are proponents of interactive and ongoing evidence synthesis. Our experience delivering interactive evidence synthesis has revealed a strong demand for customized and detailed insights on demand. This entails extracting and synthesizing impact evidence from the database on specific questions, such as “What is the impact of market-place platforms on small business growth?” This recurring need has led us to further innovate with evidence outputs to support greater access to insights using generative AI. Below are three ways we share our evidence synthesis outputs:
- An interactive impact database holds the raw coded data. Guided by a code book, we can filter, search, and access extracted summary insights and data points from the impact database. This has been a successful format for (manually) mining the database on-demand to craft impact narratives on interventions/products in specific countries on specific segments. All our evidence syntheses have an underlying impact database, resulting in a significant body of evidence on digital financial services, digital approaches to driving small business growth, digital tools to support women’s economic empowerment, and geospatial data for humanitarian action (forthcoming).
- Interactive evidence maps provide a visual overview of the entire impact evidence landscape, plotting interventions to outcomes and the direction of impact. The maps are integral for communicating a snapshot of the impact landscape and quickly discerning where evidence is concentrated. Filter buttons enable users to get more specific on other variables of interest (i.e., methodology, design and delivery mechanisms, country, segment, etc.) Alongside these interactive syntheses, Caribou has crafted dozens of targeted insights for client strategy and the public (e.g., DFS and small business).
3. A chatbot-based conversational interface queries the impact evidence database using natural language queries. This utilizes language-learning models to interpret queries and return synthesized results from the evidence base, with cited references, allowing users to, for example, type a query “Outline e-commerce products that have positively impacted women’s income in India.” This interface opens up new possibilities to engage with impact evidence, particularly when convenience and speed are prioritized and when supporting greater access to insights for those traditionally marginalized from more complex knowledge products.
What’s next
This is an exciting time to be working in evidence synthesis. With innovative new tools and processes to work with, there are more possibilities than ever to advance the efficiency, accuracy, diversity, and inclusivity of evidence synthesis. In the coming months, we will share more about our iterations with Gen AI and our approaches to deepening inclusivity and equity within the evidence synthesis process and products. We are keen to connect with users and practitioners alike to refine and perfect these approaches.
If you have any feedback or questions about our evidence synthesis approach, contact us.
Authors
