It is well known that public health issues that affect the world’s most vulnerable populations – food insecurity, maternal and child health, water- and sanitation-related disease, and resource scarcity – are inextricably linked. Where these linkages are strongest, experience on the ground has shown that community-based integrated approaches to development provide more effective and sustainable solutions over vertical, sector-based programs. But so far, there are very few comprehensive evaluations providing strong quantitative evidence of this advantage.

In general, there are few truly integrated programs being implemented globally – a preference for sector-based interventions without coordination is still the norm. Some say the primary challenge to scaling up integrated approaches like population, health, and environment (PHE) projects is the lack of evidence to convince donors why they should fund them, especially if they are only interested in health or only in environment.

Last fall, I traveled with a classmate from George Washington University’s School of Public Health to Ethiopia to fulfill the requirements of our practical experience and see how PHE projects there tackle evaluation. I worked with the PHE Ethiopia Consortium, an Ethiopian non-government organization which supports more than 40 local development organizations, 10 of which are currently combining reproductive health, environmental conservation, and sustainable livelihood components in select projects.

We spent the semester assessing the monitoring and evaluation of two of these PHE projects. I traveled to the far northern water-stressed region of Tigray to work with the Relief Society of Tigray (REST) while my colleague worked with LEM Ethiopia from Addis Ababa.

Building on Ethiopia’s Success

Ethiopians face infamous and chronic food insecurity; shrinking farm plots and low crop yields; vulnerability to climate change; poor soil quality; and extensive erosion.

These issues are due in part to a growing population of more than 87 million (the rural fertility rate is 5.5 children per woman), around 80 percent of which depend on rural subsistence agriculture.

Adoption of the PHE approach by Ethiopian development organizations and support from the government has been growing over the last decade due to its great potential for addressing these linked issues by meeting unmet demand for reproductive health services, improving farming practices, and reducing deforestation and soil loss.

But organizations are struggling to gather the data necessary to quantitatively demonstrate that these integrated programs have an enhanced value over single-sector interventions.

“It is challenging to create buy-in for the PHE approach in Ethiopia when there is very little data supporting their projects’ success stories,” said the consortium’s executive director, Negash Teklu.

While success stories are great, in order to further scale up nationally and globally, evaluations providing strong evidence of added value are needed.

Evaluating Evaluation

When I arrived in Tigray, I quickly found that the question of the adequacy of REST’s PHE evaluations was not even relevant because the monitoring and evaluation (“M&E” in development-speak) effort specific to their PHE sites was essentially non-existent.

PHE interventions are highly complex, involving multiple stakeholders across sectors. Projects are often small and rural, with a short time span and very limited funding. What funding they do have often comes from donors that primarily care about single sector results, leading to incomplete evaluation and difficulty showing the advantages to integration.

REST is made up of three sector-specific departments (health, water, and agriculture/environment) which work with a multitude of donors on sector-specific programs. PHE integration was explicitly included in Packard Foundation funding on an adolescent reproductive health program, and integration was piloted in 2005 at a few sites where REST already had other donor-funded environment and health programs operating at the same time. The existence of all components of PHE made implementing better integration between these programs natural and intuitive; however, because the integration component is not a separate donor-funded program there are no specific reporting or M&E requirements.

I pored over as many proposals, reports, M&E tools, and other documentation of REST programs as I could find, looking for data I could collate. I spoke with beneficiaries, community volunteers (health extension workers, adolescent peer educators, and agricultural development assistants), program staff, and government administration at each level of reporting, from the village to the region, to try and make sense of how REST was conducting M&E and whether any information demonstrating the value of PHE integration could be extracted.

I found that REST has the capacity to conduct rigorous evaluations through its M&E department, as evidenced by documentation for various other donor-funded programs; however, their data was representative of areas larger than individual PHE communities and therefore could not demonstrate the added value of integration.

A Global Challenge

After returning to DC to complete my studies, I wanted to see if there were lessons for Ethiopia that could be learned from other PHE programs globally.

I compared REST’s evaluation strategy and challenges to six other evaluations of PHE projects: IPOPCORM in the Philippines, CEMOPLAF in Ecuador, the Environmental Health Program in Madagascar, the Terai Arc Landscape project in Nepal, Conservation Through Public Health in Uganda, and SPREAD in Rwanda.

Three of these were actually able to provide adequate evidence of the added value of PHE integration: IPOPCORM, CEMOPLAF, and the Environmental Health Program. These programs were all “operations research,” meaning they benefited from additional resources and technical expertise designated to answer in-depth research questions and included reasonably controlled quasi-experimental studies with pretest-posttest designs.

The IPOPCORM study methods were very rigorous, and the result is the only PHE evaluation that has been published in a peer-reviewed journal, giving it the most credibility to date. CEMOPLAF collaborated with World Neighbors and benefited from technical experts who trained local staff. For the Environmental Health Program, USAID collaborated with a local NGO consortium Vohary Salama and the Madagascar Department of Demography and Social Statistics to conduct household surveys.

The challenges to conducting adequate evaluations in Ethiopia mirror the challenges experienced in all of the other global cases I examined. But with the higher capacity for planning, data collection, and analysis by their partners, these three were able to provide evidence of added value.

These examples show that quantitatively demonstrating the value of integrated development is possible. But the technical capacity and resources available to the three successful programs were higher than most PHE projects can afford or have access to.

Make Data a Priority

The assumed benefits of PHE integration are particularly appropriate today, in light of the current economic downturn and the increasing urgency of climate change. Increased evidence of the benefits for communities in ecologically threatened areas with rapidly growing populations and high unmet need for reproductive health services may provide the push needed to convince international donors and governments to move away from the traditional single-sector model and address other multi-sectoral challenges as well.

Everyone I interviewed was convinced of the added benefits of the PHE approach, they just didn’t have the capacity to prove it yet. Study design challenges, due to the complexity of PHE, play a role but the primary barrier is insufficient capacity and inadequate financial and technical resources for monitoring and evaluation.

Improving monitoring and evaluation capacity will provide the evidence needed to facilitate scaling up. In that respect, Ethiopia’s combination of an established network of PHE organizations, committed donors, and government support has the potential to serve as a role model for other countries.

I recommend that all partners involved in PHE projects make a commitment to rigorous evaluation. In Ethiopia this includes the government, the PHE Ethiopia Consortium, the international donor community, and the local implementing organizations themselves. Secondly, I recommend that adequate evaluation capacity of data collectors, data analyzers, and evaluation planners is ensured at all stages of a project’s evaluation. Once all parties commit adequate resources and capacity is addressed, the fundamental study design and methodology should be improved. But the commitment and resources must be the first priority.

Stay tuned for the continuation of this miniseries, “PHE Evaluation in Ethiopia,” on New Security Beat.