Towards Equitable Program Evaluation: Lessons from NNPHI’s Vaccine Equity Project
- By: Loren Milliken, MSW, MPH,
- Date
The development and distribution of COVID-19 vaccines marked a pivotal moment in the US’s fight against the pandemic. As states rushed to secure vaccines for their populations, the issue of vaccine equity quickly emerged as a critical concern. Vaccine equity refers to the fair and equitable distribution of vaccines to all people. COVID-19 disproportionately affected vulnerable and marginalized communities, and many communities remain vaccine hesitant for a range of important reasons. Ensuring that vaccines and evidence-based information reach these communities and respond to their concerns helps prevent unnecessary deaths. The Vaccine Equity Project sought to do just that by addressing the barriers to access to vaccines and information about public health threats experienced by communities of color, rural communities, and tribal nations.
The Vaccine Equity Project focused on increasing COVID-19 vaccine access and uptake while decreasing vaccine hesitancy in communities with high rates of unvaccinated people. The project, made possible through grant funding by the Centers for Disease Control to the National Network of Public Health Institutes (NNPHI), leverages community-based organizations working locally to meet community needs.
After a competitive request for proposal process, NNPHI funded five community partner organizations: African Family Health Organization, Project HEALINGS, The Dolores Huerta Foundation, the Inter-Tribal Council of Michigan/National Native Network, and Public Health Solutions.
The African Family Health Organization (AFAHO) is a non-profit organization based in the greater Philadelphia area whose mission is to address unmet needs among African and Caribbean immigrants and refugees who experience difficulties in accessing health and human services due to cultural, geographic, linguistic, and other barriers.
Project HEALINGS is a community coalition of multicultural partners (Lakota, Latinx, pan-Asian, Hmong, Black-African-American, LGBTQ+) led by the Chinese American Chamber of Commerce in Minnesota, focused on health and wellness among Minnesotans disenfranchised by identity and circumstance, facing disparities in healthcare access and health outcomes and disproportionately impacted by the COVID-19 pandemic.
The Dolores Huerta Foundation is a 501 (c)3 organization that serves Black, Latinx, Farmworkers, Immigrant, Indigenous Mixteco and Triqui communities, LGBTQ+, people of color and unhoused individuals in California’s Central Valley. These rural, hard to reach communities face barriers such as transportation, language access issues, lack of culturally competent health services, and misinformation and disinformation resulting in vaccine hesitancy.
Inter-Tribal Council of Michigan, Inc. is a state chartered 501 (c)3 non-profit organization consisting of twelve federally recognized Indian Tribes in Michigan, and the home of the National Native Network. Central to their mission is advocating for member tribes in the development of programs and policies which will improve the economy, education, and quality of life for Native Americans.
Public Health Solutions’ mission is to support underserved New Yorkers and their families in achieving optimal health and building pathways to reach their potential, with a particular focus on pregnant and birthing persons. As the largest public health nonprofit serving New York City, the nonprofit supports over 230 community-based organizations and provides direct services, among other health equity-focused projects.
From December 2021 through July 2023, and continuing for one more year through July 2024, each of these five community-based organizations was funded by NNPHI to implement evidence-based or promising strategies to reduce vaccine hesitancy, increase access to vaccines, and/or increase vaccine uptake in their communities.
As with vaccine equity, the evaluation of public health programs must take measures to protect against inadvertently exacerbating health disparities or inequalities. Equity in evaluation is a way to improve the effectiveness of programs and policies and build partnerships, and this project centered community-driven measures of success alongside federally-defined metrics. With a demonstrated history and a commitment to racial and health equity and community-centered approaches, Health Resources in Action (HRiA) was chosen by NNPHI to lead the project’s overarching evaluation across the five-grantee portfolio.
Nora Daly, Senior Research Associate from HRiA, stated that the goal was to “center grantee expertise and develop an evaluation responsive to each community’s context, priorities, and values.”
The HRiA’s team, led by Ilana Raskind, Senior Research Scientist, used a mixed-methods design and a developmental evaluation approach to guide and inform the Vaccine Equity Project’s evaluation.
Mixed Methods – which combines the use of qualitative and quantitative data- allows for a richer, more comprehensive understanding of a phenomenon while also having the potential to manage inherent biases that any single method may have. The team conducted 12 key informant interviews with staff from grantee organizations and their partners using a semi-structured interview guide, which aided in consistency across the five sites. The interview guide included an array of sections, including questions about changes in vaccine outreach, access, uptake, and equity in the communities served by grantee organizations, partnerships developed for this work, and the impact of culturally responsive communication strategies, amongst other vaccine equity-focused topics.
To be used as a key source of quantitative data, the team designed the Vaccine Equity Project Partner Survey to gather information regarding the relationship between grantees and their local external partners. This brief 10-minute survey was designed to document grantees’ efforts to deepen and expand their reach in local communities with a legacy of social and historical injustice.
Early in the project, NNPHI, HRiA, and the grantees highlighted the importance of collecting demographic data directly from participants via the project’s quantitative data methods, including an event tracking sheet and site-specific evaluation reports. However, once the project launched, grantees found that it was not always possible or appropriate to ask for racial and ethnic identity, given that getting people into the door of a vaccine event was already challenging due to various factors, including immigration status or lack of trust.
Taking this feedback into consideration, HRiA and NNPHI encouraged grantees to gather demographic information when and how they felt it was helpful and appropriate. HRiA’s final evaluation included a general summary of the racial, ethnic, and other sociodemographic characteristics of populations served by grantees and their partner organizations in place of specific, individual level data.
The ability to adapt to grantee needs is indicative of HRiA’s commitment to the core tenets of developmental evaluation (DE). This approach values flexibility, iteration, and continuous feedback loops. The DE approach was well-suited to the overarching assessment of the project because it allowed the HRiA team to respond in real time and make adaptations based on grantee needs and experiences.
As to be expected, the uptake of vaccines was an important metric to measure. Four out of five sites hosted vaccine clinics, delivering over 120 events and providing over 5,500 shots, with all five sites engaging in communication campaigns and provider trainings designed to build trust and reduce vaccine hesitancy. However, if vaccine uptake were to be the primary focus of a vaccine equity project, it would be short-sighted.
As Raskind states, “If shots in the arm were the only focus of the evaluation, we would miss the grantees’ time, energy, and commitment to building trust and developing meaningful community partnerships that will last beyond the project period.”
The clinics successfully reached people, but what is notable, yet more difficult to capture than “shots in arms,” is how grantees increased trust and built relationships, a key measure for future public health interventions. One measure that was indicative of moving the needle forward was the increase in collaboration between grantees and local partners. According to the Vaccine Equity Project Partner Survey, most external partners (N=34) set project goals and made decisions about events and activities with their associated grantees. This capacity building may be leveraged during public health events or for other preventative healthcare needs.
Equity in evaluation is an ethical imperative and a way to improve the effectiveness of programs and policies. By identifying and naming inequities, evaluations can contribute to more equitable outcomes and better serve the needs of diverse populations in meaningful ways that are responsive to community members. Equity in evaluation is essential to ensure that all communities of focus, particularly those who have been isolated, marginalized or historically underserved, have an equal opportunity to have their narratives included and benefit from the evaluation findings.
Next up – NNPHI and HRiA will be working with Spitfire Strategies and Asian Media Access, in partnership with our CBO partners to create a catalogue of promising practices based on the findings of the Vaccine Equity Project work and the Guiding Principles that have been identified across all of the implementation sites. Stay tuned for more on NNPHI’s Vaccine Equity Project!
Subscribe To Our Communications