By Dom Margiotta, Program Associate and Felipe Rangel, Program Assistant at Social Impact

A new study by Social Impact and Social Solutions International seeks to better understand how USAID has applied developmental evaluation (DE) in a health context by documenting lessons learned from: 1)  USAID/Tanzania’s DE for their Boresha Afya program, hereafter referred to as “Boresha Afya DE”, and 2) USAID/Indonesia’s DE for their Jalin project, hereafter referred to as “Jalin DE.” These programs offer valuable knowledge for planning, designing, and implementing DEs for health in the future, including benefits, constraints, approaches to ensuring better local government buy-in, and embedding evaluators.

Developmental evaluation is an evaluation approach that supports complex and innovative programs by embedding evaluators in project teams to provide real time feedback and assist them with managing adaptively. DE focuses on adaptation and flexibility, making it well-suited for complex environments, innovative programs, and untested approaches. To learn more, see USAID Developmental Evaluation Pilot Activity (DEPA-MERL)’s resources, the Jalin DE guide for conducting DE remotely, and CIRCLE’s seven steps to setting up a developmental evaluation.

Benefits: Health programs demand an approach that accommodates multisectoral collaborating, learning, and adapting (CLA) and can respond to their complex operating context. DE is a valuable way to make CLA happen, as Elizabeth Macgregor-Skinner, Director of USAID’s Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) Project, notes in her blog on How DE makes real-time learning happen. Given how DE inherently emphasizes rapid and robust stakeholder feedback loops, it is well-positioned to make evidence-based adaptations and course corrections in real time, and thus has proven effective in meeting these needs. In fact, stakeholder feedback loops that Jalin DE developed for the Indonesia Ministry of Health (MOH) compelled USAID to select them as one of their seven 2021 CLA Case Competition winners (out of more than 80 entrants from 71 countries).

Constraints: Compared to traditional evaluation approaches, implementing a well-run DE depends on embedding evaluators, which requires relationship building. As such, DEs require substantially more time, financial, and human investment. Furthermore, DEs must launch as close as possible to when their program or project does to maximize synchronicity and overall success. That said, study respondents from Boresha Afya and Jalin DEs agree that taking time to actively help stakeholders understand DE’s objectives, methods, and engagement practices can help alleviate these constraints and dispel the perception of DE as a judgmental watchdog.

Local Government Buy-in: In Tanzania, the President’s Office for Regional Administration and Local Government and council health management teams valued Boresha Afya DE’s ability to collect and react to real-time data.

In Indonesia, Jalin DE’s’ approach complemented the MOH Family Health Directorate’s research agenda, which involved testing and scaling pilot programs in hospital mentoring, digital indicator data entry, and a maternal and perinatal death notification application.

This supports findings from an independent DEPA-MERL study designed to facilitate learning on and effective implementation of DE based on three completed DE’s in the USAID context. That study found 60 percent of stakeholders preferred DE over traditional evaluation, and 80 percent of stakeholders would recommend DE to other organizations.

Embedding evaluators: Boresha Afya and Jalin DEs support the notion that embedded developmental evaluators are integral to DE success. However, with health systems that have many stakeholders across many areas, DE planners face extra pressure in deciding where to embed evaluators.

Boresha Afya DE embedded zonal developmental evaluators in regional health teams alongside central investigators and specialists. Jalin DE embedded in Jalin headquarters and made monthly trips to provincial and district health offices. In its final year, Jalin DE embedded evaluators in the MOH.

While this study found compelling lessons learned for health DEs based on Boresha Afya and Jalin DEs, as Dr. Ashwin Budden of USAID’s CIRCLE Project notes in his blog on setting up a DE,  there are important steps that both donors and implementers alike should consider before taking on a DE. One involves socializing stakeholders to DE, which one can do by introducing CLA and complexity-aware monitoring (CAM) principles at the outset and thereby setting the stage for later capacity building. Another involves USAID helping programs and stakeholders understand the ways they inherently conduct CLA and CAM (intentionally or not). This will likely help stakeholders understand the ways DE can support their work, and in turn sets the stage for a DE that stakeholders believe is integral to the program or project rather than invasive or out-of-sync.

About the Authors: Dom Margiotta is a Program Associate at Social Impact. He has supported DEPA-MERL since 2017 and Jalin DE since 2020, co-authoring and delivering many thought leadership and knowledge products for them (including a guide to conducting DEs remotely and webinar on conducting DEs remotely). Felipe Rangel is a Program Assistant at Social Impact and has supported DEPA-MERL and Jalin DE since March 2021. Both Dom and Felipe contributed to Jalin DE’s study of USAID’s DE involvement over the last 10 years.

Acknowledgments: This study would not have been possible without the support of USAID/Indonesia, USAID/Tanzania, USAID’s Research Policy Division, USAID’s MCHN Office, and the USAID Developmental Evaluation Pilot Activity (DEPA-MERL).

Cover Photo Credit: Oscar Siagian, USAID Jalin