Since the integrated population, health, and environment (PHE) approach is relatively new in international development, donors, partners, and implementers want to know how it’s improving people’s lives. In the PHE community, we believe that combining efforts to address natural resource management, reproductive health, and livelihoods is making a difference in places where rapid population growth combines with poverty and environment degradation. But to know for sure and be able to convince others, we need to have data to support those beliefs.

As Laurel Hamilton explained in the first post in this mini-series on PHE evaluation in Ethiopia, the first impact evaluations of PHE projects in Ethiopia have just started. In part because of its unique development challenges but also because of its strong network of grassroots development organizations, Ethiopia is one of the most active countries in terms of number and history of PHE implementers. But many of the local organizations do not yet have strong monitoring and evaluation (M&E) systems for tracking their results.

In recognition of this and to help with the unique multi-sectoral nature of PHE projects, The David and Lucile Packard Foundation, USAID, and the PHE Ethiopia Consortium – an Ethiopian-run network of more than 40 development organizations and PHE implementers – have partnered to create a position for a “PHE Advisor” in Ethiopia through the USAID Global Health Fellows II Program.

Over the last year, I’ve been working as this advisor, collaborating with the PHE Ethiopia Consortium and its members. We are focused on answering five questions to learn more about the difference PHE projects are making on the ground.

What Data Is Important to Collect?

PHE projects have livelihood (e.g., helping to setup vegetable and honey production ventures) and agricultural (e.g., giving farmers access to new seed varieties) components in addition to population, health, and environment interventions. Organizations also want to track the secondary effects of these various interventions, such as how their projects are empowering women and young people.

With this diversity of components, implementers can find themselves swimming in a sea of indicators without a clear sense of which to track. Oftentimes, they simply pick the easiest: output indicators. This makes life easier in the short run when they only have to track numbers of people trained, seedlings planted, or condoms distributed. However, when someone asks what difference their project made, they aren’t able to provide a strong answer because they don’t have an idea of what all these outputs add up to.

To address this, we are working with organizations to develop M&E plans and pick indicators that will better reflect the true difference they are making in peoples’ lives. (Laurel and her colleague from George Washington University helped in this effort, by working with REST and LEM Ethiopia to learn about their organization-specific needs last summer.) The PHE Ethiopia Consortium is also developing a shared M&E framework at the national level. Once established, members of the consortium will report on these indicators on a semi-annual basis, hopefully creating a much broader and more uniform set of data to compare.

How Do You Collect the Data?

Even if we’ve established what data is needed, getting it can be a challenge.

Collection of the most interesting data – what changes have been made in people’s lives – often requires doing a survey or having another way to find out knowledge, attitudes, and practices of a representative group. Many organizations aren’t able to do regular surveys because they lack time, money, or capacity, so they have focused on easier data collection methods, like counting people who signed the attendance sheet at a training session. Unfortunately, these methods don’t tell you what happened to those participants as a result of attending the training.

It can also be difficult to determine denominators for percentages. For example, in order to calculate the percent of a community using family planning, you need to know the number of married women of reproductive age in the project area. For many areas, that data just doesn’t exist.

Another challenge is that most PHE organizations in Ethiopia do not directly provide health services. Many organizations provide counseling, education, and a limited range of contraceptive methods and then have a referral linkage with the government health system for other services. A benefit of working with the government is that PHE organizations can reach more people and share data. However, there are also some challenges, as the data may be for a whole district, instead of the specific community where they’re working. Another is data quality; there are sometimes differences in how indicators are defined or how the data is collected.

The PHE Ethiopia Consortium and I are trying to address these challenges by using indicator reference sheets and encouraging organizations to involve stakeholders in the discussions around selecting indicators so there can be a shared understanding of what data to collect and why. We are also trying to link our shared health indicators with the government’s official Ethiopian Health Management Information System. This will also help us to make comparisons between PHE and non-PHE sites. The government system is currently being rolled out across the country. During this process, there may be a few bumps as everyone learns how to use the system; however, when it is in place and up and running, the data quality should be much better and more uniform.

How Do You Keep Track of the Data?

There is a wide range of experience with M&E across PHE implementers in Ethiopia. There are more than 40 organizations that are members of the PHE Ethiopia Consortium alone. Some have research departments and databases; others collect data in trip reports. If they want to aggregate data over time or sites, they need to open all the reports and add up the numbers. They also have to hope that whoever recorded the data included all the important information and saved all their reports!

With the PHE Ethiopia Consortium, we are developing a web-based M&E system which will be supported by the consortium’s staff. Each organization will have their own account where they can include the indicators with data from their different projects. They’ll also be able to generate reports and maps. We are piloting the system with five organizations. After they give us their feedback, we will refine it and then open it up to all the consortium’s members.

How Do You Make Sense of the Data?

Attribution for PHE projects in Ethiopia is tricky: You need to be able to determine if the results of a particular project are because of the PHE implementers’ work, other factors, or a combination of both. And because projects are often closely linked with government efforts, it becomes hard to separate the contributions of different players.

To address this we are paying particular attention to attribution during M&E trainings so implementers can better understand when and how to claim credit for their contributions to indicators like an increase in the number of family planning users or household income. We are also developing a “PHE Implementation Manual” that will provide guidance on how to make sense of data and use it for decision-making.

The PHE Ethiopia Consortium recently completed a rigorous impact evaluation of the Guraghe People’s Self-Help Development Organization (GPSDO). Conducted in conjunction with Jimma University and funding from MEASURE Evaluation, the evaluation found that the PHE approach leads to more positive changes in many fertility behaviors than a reproductive-health-only approach. The consortium hopes to conduct more research like this and is partnering with Johns Hopkins University Bloomberg School of Public Health, Jimma University, and Mekele University to have interns develop case studies and possibly conduct operations research in the future.

How Do You Communicate the Data to Others?

Most organizations find it difficult to effectively communicate their results with stakeholders – from donors, to government, to community members. It can be especially complicated to communicate PHE results because of its integrated nature: It can be clear different activities happened but it’s not always clear how they relate to each other or whether individual community members are receiving the full range of interventions.

This is another area where the PHE Ethiopia Consortium can be a great use to its member organizations. Their staff have experience working with the media and are connected to the international PHE network, where implementers are constantly sharing their experiences about communicating results. The Packard Foundation, Population Action International, Population Reference Bureau, and the Wilson Center have all provided support to PHE implementers to share their experiences.

The PHE Ethiopia Consortium has used this funding and experience to publish policy briefs and stories about their member organizations (Spotlights). They also hold quarterly online forums, present the PHE method to journalists, and host events and sharing visits. And they work to reach decision-makers through incorporating PHE messages into international days of action such as Earth Day and World Population Day. Together, we are currently developing a suite of communication materials to complement existing publications, document good practices, and champion stories from Ethiopia.

—

The Packard Foundation and USAID have been strong champions and supporters for population, health, and environment projects in Ethiopia. Their support, and the work of countless Ethiopian implementing organizations, has enabled PHE to take root and make positive changes in the lives of thousands of people in over 30 districts. The PHE Ethiopia Consortium has also been able to grow and provide needed support to its members.

But for us learn the most from these experiences the PHE community needs to deepen its focus on these five questions about data collection. By evaluating existing investments, we can move PHE forward and show that it is a promising tool for addressing some of the most difficult development, health, and environmental challenges today.

Stay tuned for the conclusion of this miniseries, “PHE Evaluation in Ethiopia,” on New Security Beat.