Navigating the AI Revolution in Evaluation

With over 700 attendees worldwide and more than 200 engaging sessions to choose from over three days, the European Evaluation Society’s 15th Biennial Conference in Rimini, Italy, was a great place to take the pulse on emerging trends within the measurement and evaluation community. Elise Montano, Director of Measurement and Impact at Caribou, was there and, in this blog, highlights her key insights from the week, including the acceleration of AI in evaluation, the growing demand for evidence synthesis, and the need for adaptive measurement and impact evaluation.

Much of the evaluation process we’re familiar with — design, data collection, transcription, analysis, validation, and visualization — can already be handled by AI. AI is showing efficiency gains over human-led processes. Rather than resisting this shift, the evaluation community should find ways to collaborate with AI to make work more efficient. AI adoption is gaining momentum from major development institutions like the World Bank, UNICEF, UNDP, IDB, and ADB down to individual consultants.

However, the voices concerned about AI use in evaluation were well represented. These concerns included a loss of creativity, data privacy, the potential for bias, widening digital skills gaps, and an over-reliance on AI for critical thinking. These illustrate some of the significant challenges that will continue to shape how we embrace AI in the coming months and years.

Caribou has long recognized the power of impact evidence synthesis to support strategic decisions — and we are not alone. Many organizations at EES are investing in more robust evidence synthesis to understand what types of interventions are generating impacts, why, and how.

However, keeping tools like evidence maps agile and up-to-date is labor-intensive. Organizations are increasingly turning to machine learning and AI to automate extraction, organization, coding, and data mining processes. Tools, like AIDA allow users to query the UNDP’s vast evaluation database to identify impact results and generate insights based on specific user queries. While there is still a long way to go to address issues of bias and to understand the traceability and explainability of results, it is clear that layering Gen AI on top of evidence synthesis is the next step in evidence synthesis’s evolution.

Evaluation has progressed significantly since experimental RCTs dominated the landscape. Theory-based approaches and mixed-methods evaluations are now widely embraced. Yet, there’s still concern that current methods aren’t agile enough for innovation-heavy programs that require real-time, actionable insights to pivot strategies and optimize outcomes.

At Caribou, we’ve found ways to navigate this tension, using principles from user-centered design, lean data collection, and our approach to modular evaluations to help balance the needs of accountability and learning. But it’s clear that evaluators need more tools to balance innovation and learning with rigor and accountability. Looking ahead, there’s potential to incorporate good practices from generative AI and evidence synthesis into traditional evaluation methods, making evaluations more robust and efficient.

Our Measurement and Impact team is committed to advancing the field by investing in evidence synthesis, modular evaluations, and AI experimentation. We’re constantly exploring how these approaches can enhance the rigor and efficiency of our work, ultimately increasing our impact.

Collaboration is the best way to navigate the evolving landscape of evaluation. If you’re equally passionate about leveraging cutting-edge practices to shape the future of evaluation, contact us.