Resilience Corps: Assessing a Pilot Job-Training Program to Increase Community Preparedness to Future Emergencies

A Formal Proposal for An Outcome Evaluation

2/2 I. Specific Aims

During the early months of the COVID-19 pandemic, government leaders in cities across the U.S. struggled to adequately respond to a disease that rapidly spread through resident populations and inflicted devastating economic damage. Some non-governmental organizations stepped in to provide critical assistance, either directly aiding governmental efforts or working with public officials to launch their own support programs.

In New Orleans, the nonprofit organization Resilience Force (RF) designed one such program, the Resilience Corps (RC), to support immediate and long-term recovery for communities most impacted by the pandemic and, ultimately, promote economic and social justice in the face of future crises. The key assumption behind this program’s impact theory is that employing residents of target communities as public health aides will increase other residents’ receptivity to information, encourage best health practices, and expand utilization of critical social services. If successful, the two-year pilot program will serve as proof of concept that cities and states can apply to other communities in need of culturally-competent crisis support services.

This proposed evaluation of the RC has important ramifications for a host of external and internal stakeholders – from policymakers to nonprofit leaders to philanthropic organizations. For one, the study’s findings will inform the work of RF, which operates similar initiatives in cities across the U.S. The results of this evaluation will offer critical insights into the assumptions underpinning RF’s work in vulnerable communities. Given the breadth of public health dangers, from COVID-19 to diabetes to hurricanes, these results will also provide valuable information for several Louisiana and New Orleans agencies, not just the state and local health departments that partnered with RF to deploy the RC pilot. Finally, philanthropic partners like the Rockefeller Foundation and the Ford Foundation provided critical support to help launch this program. These capital partners support such initiatives in hopes they will meaningfully boost public health and social justice. For these organizations, this report can inform if and how they choose to support future applicants.

This proposal lays out a plan to evaluate the RC program’s effects on two of its primary stated goals: increasing vaccination rates and expanding awareness of pandemic-related social services in the New Orleans areas hardest hit by the pandemic. Using a quasi-experimental design, this study will compare a group of the RC-designated census tracts and a group of similar tracts that have not received the intervention. It pursues a classic 2x2 design and an interrupted time series design to answer two key research questions that align with the program’s goals. To assess the two stated goals above, we have proposed the following research questions, which rely on two validated and reliable measures:

Do census tracts in New Orleans targeted by the RC post a larger increase in first-dose COVID-19 vaccination rates, compared to similar tracts not exposed to the RC, from May 1, 2021 to October 1, 2021?

Do the RC-designated census tracts experience a larger increase in unemployed residents receiving unemployment insurance, compared to the tracts not exposed to the RC, across the first eight months of the RC intervention between October 1, 2020 and May 1, 2021?

4/4 II. Background and Significance

Statement of the Problem

Government officials face an array of challenges in public health administration, including a persistent lack of trust among members of the public. Trust in public health officials is essential in building avenues of communication to effectively disseminate information about important safety measures and available supportive services. In one 2017 study, researchers found that higher trust levels in public health officials were not just positively correlated with higher vaccination rates, but they predicted vaccine behavior (Ward, 2017).

COVID-19 shined a spotlight on the pernicious effects of low trust in public health figures. During the pandemic, residents in cities across the U.S. faced heightened risks, as mistrust combined with misinformation to dent confidence in public health messaging. Misinformation and poor messaging around available services pose even graver threats in communities that were already vulnerable during public health crises, due to pre-existing social and economic inequalities. Indeed, during COVID-19, low-income communities of color suffered disproportionately (Centers for Disease Control and Prevention, 2020).

Those trends played out in cities across the U.S., but especially in New Orleans, Louisiana. During the first two months of the pandemic, New Orleans’ per-capita death rate was the highest in the country. However, the pain was not felt equally. As of June 2020, Black residents accounted for 88% of total deaths due to COVID-19 complications (Williams, 2020). Given New Orleans’ racial and socio-economic makeup, with 60% of residents identifying as Black or African-American and 24% of its population living in poverty, this is particularly concerning. It begs the question: how do we get the right information to vulnerable communities who do not trust our public authorities?

Program

Government leaders have attempted to combat mistrust and lack of public awareness through marketing campaigns and enlisting “ambassadors” – people whom members of at-risk populations may find reputable and respectable, like celebrities. However, the pandemic has shown those may not be the most effective methods. In their place, governments and nonprofits have spearheaded a range of alternatives, including the RC. Launched in October 2020 by RF, the RC is a multi-pronged program to support low-income and racial-minority New Orleans residents. The RC program aims to introduce a new model for responding to public health and safety crises that relies on the trust people have built with neighbors in their communities. The model is centered around training local residents who lost employment due to COVID-19 as aides to support public health messaging and vaccination efforts. The program then deploys them into their own communities to engage other residents on sensitive topics such as vaccinations and government support services. The RC also charts a path for how the City of New Orleans can invest in year-round resilience work and create jobs in neighborhoods on the frontlines of these disasters that grow more frequent and intense every year.

Program Theory

The RC program is currently a two-year pilot produced in partnership with the City of New Orleans and the State of Louisiana (Funes, 2021). The RC also relies on support from private sector entities, including the Rockefeller Foundation (Rockefeller Foundation, 2021). The program is rooted in the belief that, if New Orleans can build this strong corps, the city’s residents – particularly socially marginalized groups in historically disinvested areas – will better withstand and rebuild from future disasters (Resilience Force, 2020). In that sense, the RC embraces the notion of resilience championed by University of Colorado Sociologist Kathleen Tierney – “preexisting, planned, and naturally emerging activities that make societies and communities better able to cope, adapt, and sustain themselves when disasters occur, and also to develop ways of recovering following such events” (Tierney, 2014, p. 5).

We illustrate the RC’s logic model in Figure A of the Appendix. It can be best interpreted by referring first to the RC’s impact theory on the right-hand side of the model. The program's proximal goals (outcomes) include increasing trust around public health, decreasing vaccine hesitancy/increasing vaccination rates, and enhancing people’s ability to weather public health and safety crises through community support. Its longer-term aspirations (impacts) center around economic and social justice for New Orleans' most vulnerable residents in the face of future emergencies and natural disasters. Moving leftward on the model, the RC employs a multitude of activities, ranging from building and training a resilience workforce to door-to-door canvassing and advocacy, in order to achieve its outcomes and impacts. These activities are tied directly to corresponding outputs. These two elements, stemming from the RC’s core resources, are the main pillars of the program’s process theory. 

There are several causal assumptions that thread through the program’s process theory and impact theory, which are outlined above the logic model. A key process theory assumption is that recently laid-off individuals are willing to put themselves at risk on the frontlines in neighborhoods with disproportionately higher rates of COVID-19 to fulfill door-to-door canvassing – a crucial programmatic activity. The initiative also assumes that the RC’s targeted population wants to be reached at their home; that is, households will answer the door during a pandemic. There are also a number of causal assumptions when examining the RC’s impact theory. The program assumes that the knowledge and support imparted by the RC will lead to sustained changed behavior in residents after the emergency situation stops feeling like an urgent crisis (i.e., are people still vigilant two years into a pandemic or two years into disaster recovery efforts?).

The most crucial assumption baked into the RC’s impact theory is that targeted populations are more receptive to the RC staff, who are their neighbors, compared to credentialed public health officials from organizations like the Centers for Disease Control and Prevention (CDC) or Federal Emergency Management Agency (FEMA) who may otherwise intervene. This is different from when government authorities or even non-profit organizations set up local offices in communities they aim to serve. It is not always true that those employees will be familiar, or trustworthy, to residents in their target population. While studies have shown that people tend to trust those who look familiar, an evaluation of this program could inform new models of public engagement that employ a place-based approach to disseminating crucial information (Devitt, 2018).

Literature Review

After assessing existing literature on models adjacent to the RC, we concluded that research supports using community-based interventions to change behavior in the target population. In a study of community-based injury prevention interventions, researchers found that the community-based designs enhanced the programs’ impacts. “Interventions are more effective when they are integrated into the community and when approaches are tailored to address unique community characteristics such as ethnicity or socioeconomic status,” the study authors wrote (Klassen et al., 2000, p. 85). However, researchers have not extensively studied the impact of locally-hired community-based interventions, which is at the crux of the RC model.

We can look to literature surrounding mutual aid groups to inform the impact of employing neighbors to help their neighbors. One of the RC model’s strengths is that it proposes to capture individuals who are not likely to engage with typical community-based organizations, but would be open to being engaged by their neighbors. Scholars define mutual aid groups as “self-organizing groups where people come together to address a shared health or social issue through mutual support” (Mao et al., p. 1083). Mutual aid groups face numerous challenges, but one of the greatest is sustaining the networks they build in response to disasters. That is because these units are often volunteer based and without compensation. As American sociologist Allen Barton notes, neighborly support dwindles as resources and personal drive diminish, and state interventions emerge (Barton, 1970).

In light of this existing research, this evaluation of the RC’s model is significant for several reasons. The RC differs in critical ways from most community-based initiatives and traditional mutual aid structures. The RC members are paid, and the program leaders partner directly with local and state government agencies. By studying the RC model through this evaluation, we can attempt to fill these critical gaps in the literature.

Research Questions

This proposal aims to boost knowledge around this particular community resilience initiative in New Orleans through an outcome evaluation, focusing on census tracts that the RC members will canvas. The study’s insights will inform whether or not to continue the initiative in New Orleans, as well as how RF can improve their operations in other cities. The evaluation thus focuses on two central RC program goals and corresponding research questions.

Program Goal #1: Increase COVID-19 vaccination rates in New Orleans areas targeted by the RC.

Research Question #1: Do census tracts in New Orleans targeted by the RC post a larger increase in first-dose COVID-19 vaccination rates, compared to similar tracts not exposed to the RC, from May 1, 2021 to October 1, 2021?

Program Goal #2: Increase awareness and understanding of COVID-19 social services available to the New Orleans residents in these neighborhoods.

Research Question #2: Do the RC-designated census tracts experience a larger increase in unemployed residents receiving unemployment insurance, compared to the tracts not exposed to the RC, across the first eight months of the RC intervention between October 1, 2020 and May 1, 2021?

We hypothesize that residents in the RC-designated census tracts will post higher vaccination rates and a higher share of unemployed residents who receive unemployment insurance, compared to similar census tracts that did not receive the intervention.

III. Research Design and Methods

3/3 Process Evaluation / Implementation Analysis

To validate the results of this impact evaluation and build an important benchmark for that work, we need to first conduct a process evaluation of the RC program. If we can challenge and better inform internal assumptions underpinning the process theory behind the RC’s Logic Model (See Figure A in Appendix), we can rule out flawed process theory as a cause of any outcome findings. As evaluators, we also want to confirm when an intervention has actually started in order to better gauge if and when the program is ready for an impact evaluation. We wouldn’t want to run an outcome evaluation at a premature juncture. External stakeholders, such as funders of the initiative like the Rockefeller Foundation, are also interested in knowing RF is implementing the RC as agreed upon. If we further demonstrate RF implemented the program as planned, then we can also rule out flawed implementation as a cause of any findings in our outcome evaluation.

Through the process evaluation, we will aim to understand the degree of fidelity between the planned RC program and its implementation. For example, we will inquire into the depth and coverage of the target population. Is the RC providing outreach to all residents in the program’s targeted group of census tracts, or is it just targeting homes that are more receptive (e.g., households opening their door on first contact)? How often do RC members approach residents and where do they make contact, if not at residents’ homes?

We will also evaluate the actual extent of the intervention’s implementation. Is the RC successfully recruiting recently laid-off workers affected by COVID-19, or hiring more broadly? Additionally, we will prioritize understanding obstacles in the field (such as how a public health emergency could restrict door-to-door canvassing success) and facilitating factors (such as how recruiting an area resident onto the RC staff could more rapidly help the RC build community trust). Beyond helping to ascertain whether the program in practice measures up to the program’s design, answers to the previous questions will help explicate the process theory.

To conduct this process evaluation, we will use varying methods. Prior to the RC’s official launch date, we will undergo a comprehensive document review to better understand the process theory (e.g. reviewing planning documents and funding applications). We will also periodically review documents to stay updated on the implementation process (e.g., reviewing meeting minutes and progress reports). Additionally, we will source primary qualitative data through site visits (e.g., shadowing the RC members in the field) and via key informant interviews (e.g., debriefing with the RC units after a certain period of time), whereupon we will compare our findings to the program’s original proposition.

Design

6/6 Research Design

This evaluation is prospective, beginning in June 2020, when RF assigned this evaluation team to study the specific census tracts the organization identified for the RC pilot program. For this outcome evaluation, we propose a quasi-experimental design, assessing treatment and comparison groups of New Orleans areas on two relevant measures – first-dose COVID-19 vaccination rates and the share of unemployed residents receiving jobless claims. We will strengthen the design by assessing the changes across the treatment and comparison groups, before and after the RC officially deploys corps members into targeted communities. To measure the program’s impact on COVID-19 vaccination rates, we will conduct a single pre-test, on May 1, 2021 (11 days after U.S. public health officials made vaccines available to every adult on April 19), and a single post-test, six months after, on October 1, 2021. In that case, we introduce two counterfactuals – the pre-test and the comparison group.

For the share of unemployed individuals receiving UI benefits (“UI Provision Ratio”), we will utilize an interrupted time series design with 64 total data points – 32 in the pre-test period (March 1, 2020 – October 1, 2020) and 32 in the post-test period (October 1, 2020 – May 1, 2021) – with each data point constituting a week’s average. For this second measure, we once again introduce two counterfactuals – the pre-test period and the comparison group. We chose to use interrupted time series data for this measure because of the timeline of UI claims. Individuals may receive these benefits for up to 26 weeks, and may temporarily disengage from the program if they find sufficiently attractive work for a period of time before resuming their case for UI. If we simply analyzed a count of cases every week, we would risk overcounting people. Comparing UI Provision Ratio data over the time frame before the start of the RC intervention also provides a gauge to measure the strength of our counterfactual; if the trend lines mirror each other in both groups, we will be more confident in the extent to which the comparison group is comparable to the treatment group. For both measures, the difference between the differences in the pre- and post-test periods for both groups will serve as the observed program effect. We elucidate the treatment and comparison groups below and detail them further in the “Sample” section.

The treatment group will include census tracts in New Orleans targeted by the RC, while the comparison group will feature census tracts matched on similar characteristics, where RF has not deployed RC members. Based on our conversations with lead project coordinators at RF, we understand that this program targeted census tracts in the city with 1) disproportionately high positivity rates for COVID-19 (above a certain threshold determined by RF) and 2) disproportionately high unemployment rates (above a threshold determined by RF).

We have pursued this quasi-experimental approach because it is the strongest research design possible within our practical, ethical, and financial constraints. Employing a comparison group will guard against key threats to internal validity, boost the rigor of our evaluation, and reduce the possibility that factors other than the RC intervention produced the observed effects. Integrating an interrupted time series design with multiple data points in the pre-test period (for one of our measures) will help establish comparability at the baseline, reinforcing our matching criteria to build the comparison group. This will further increase our confidence in attributing our findings to the RC program itself.

We did not pursue a randomized control trial design, because RF would not agree to randomly designate the RC and non-RC areas, given potential ethical considerations and additional logistical hurdles. Indeed, RF explicitly targeted the areas “hardest hit” by the pandemic, both economically and from a public health perspective, in partnership with city and state authorities (Resilience Corps). If tracts differ, even slightly, in positivity and unemployment rates, a clear hierarchy of need will exist. It will thus be hard to justify randomly assigning census tracts to receive the RC intervention without sacrificing the mission of the program.

We also considered a different type of quasi-experimental design, featuring a treatment group of the RC-designated tracts in New Orleans and a comparison group of census tracts in another city without the RC. However, we would face significant challenges pinpointing a similar-enough city based on our metrics of interest. Cities also possess very different public health infrastructure and UI programs that would make our outcomes difficult to assess.

Finally, we also deliberated a reflexive, single sample, pre-post design, but nixed that plan, because a strong comparison or control group is critical to control potential outside effects that impact different areas of New Orleans over time, particularly given the fast-moving dynamics around COVID-19.

Below, we have detailed the most common threats to internal validity, delineating those that do not apply in our chosen research design, those that this design mitigates against, and those that this design still fails to account for.

Internal Validity

First, it is important to note that practice effects – including the Hawthorne effect of being studied, the classic practice effect (how taking a survey or test impacts later performance on it), and the testing-treatment interaction effect (effect of a pre-test on the intervention itself) – are not applicable in this research design. We plan to use secondary data, working with aggregated statistics across census tracts. Thus, no potential testing biases are present.

The other eight common threats to internal validity do apply, and this research design will mitigate those to varying degrees. Using a carefully curated comparison group in our two models (2x2 mixed design and an interrupted time series design) controls for three significant threats to internal validity: outside effects, maturational trends, and regression to the mean.

Starting with outside effects, it is possible for New Orleans residents living in both the treatment census tracts and comparison census tracts to access other COVID-19 educational programs, outside of the RC. Perhaps, for example, local libraries launch outreach initiatives to provide additional COVID-19 resources. Even so, since both groups are exposed to these “outside” programs, we should counter this threat in our chosen designs. Furthermore, government policies can impact neighborhoods over time, but, given those are dictated at the city, state, or national level, they will likely affect these areas similarly. Even so, given the possibility of additional effects of these policies or programs at a more granular census tract level, we will use shadow controls by consulting with experts in the New Orleans Health Department to gain insights they have on any outside effects that could skew the RC’s results.

Maturational effects could be an additional threat to internal validity in this evaluation, but we will also reduce the risk through the comparison group. Secular economic trends or immigration into New Orleans, for example, could influence relevant outcomes in census tracts over the course of the program period. However, with two groups of tracts matched on critical criteria, we will likely account for most of these factors. By using an interrupted time series design and including multiple data points in the pre- and post-period for one of our measures, we will further curb the risk of missing maturational trends by compensating for seasonality and any cyclical effects around unemployment.

Finally, the comparison group further helps this design control against regression to the mean. With two groups of census tracts matched on similar metrics (and we expect pre-test scores for both vaccination rates and the share of unemployed residents receiving benefits to be roughly comparable), both are just as likely to regress to the mean. Once again, the interrupted time series design for the unemployment measure will further reduce the risk as well.

The research design also addresses concerns around instrumentation and attrition. Instrumentation is a threat when the particular device used to measure the effects of the intervention changes. Since we will use secondary data for our two research questions, this could be an issue. However, in this case, we will use well-validated and reliable measures for COVID-19 vaccination rates and UI applications, both of which are administered by Louisiana state governmental agencies. For both measures, we do not expect the processes for calculating these figures to shift across the pre- and post-periods, and we will also check in regularly with the stewards of these secondary data sources.

Attrition is also not a particular concern given we will not link responses at the individual level. Instead, we will analyze data at the census tract level by aggregating and comparing secondary statistics across the RC-designated census tracts and non-RC-designated tracts. By using publicly-available secondary data, we don’t need to worry about administering assessments or about any participants dropping out of the study. In this context, we will see attrition if RF halted the RC’s work in one or more of the census tracts analyzed, or if all of the households in a census tract refused the RC’s services. Given the relatively short time period for the evaluation, we don’t expect that to happen. Furthermore, it is also important to note here that we could have analyzed the RC based on the outcomes of actual RC staff members themselves recruited to administer the intervention in these select New Orleans areas. In that case, attrition would have been a greater concern if a significant portion of the corps itself did not choose to stick with the program. However, that is outside the scope of this evaluation.

Finally, based on the proposed research design, compensation is not a significant threat to internal validity in this evaluation. Compensation occurs when individuals or entities in the comparison group act differently than those in the program group would have without the program. Once again, because we plan to aggregate secondary data at the census tract level, we are less concerned around individual behavior. (We also do not interact with members of either the program or comparison group at any point during the evaluation). Additionally, there is no formal RC program waitlist to let residents of other areas know they were not selected, and residents across New Orleans’ census tracts can access COVID-19 information in various ways. So, residents in non-RC census tracts will have little incentive to act differently just based solely on the program’s absence in their area.

Despite the merits of this research design, it’s not a panacea. Of all the threats to internal validity, selection bias is the greatest concern in this study, given that the RC did not randomly assign census tracts to the intervention. RC leaders used specific criteria to select the areas that will receive the program, which could inherently bias the study’s findings. In other words, it is possible that tracts in this intervention group were, indeed, different even before the program started. Though, we will randomly select the comparison group census tracts out of a pool of comparable tracts to minimize selection bias here (explained further in the “Sample” section).

While evaluators can never completely control for selection bias in quasi-experimental designs, we feel confident that we will minimize this threat because the RC-designated tracts in the program group and non-RC-designated tracts in the comparison group will match across several relevant criteria. Thus, the comparison tracts almost represent a waitlist group. If RF had more resources, they would have likely selected these areas to receive the intervention as well.

This program will also still be vulnerable to contamination, given the high likelihood of knowledge spillover between residents in the RC-designated census tracts and non-RC tracts. The boundaries between census tracts are rigid, but it’s very likely that residents in an RC-designated census tract might transmit knowledge learned from the RC members/door-knockers to members of their social network in non-RC-designated census tracts. We could address contamination if we geographically separated the program and comparison groups, but mitigating contamination through that type of design does not outweigh the other challenges it would present.

External Validity

In assessing external validity, it is critical to underscore that this evaluation only examines the RC model (increasing community preparedness to emergencies and disasters) within the contexts of New Orleans and COVID-19. Thus, this design specifically highlights the effects of the RC on certain vulnerable, low-income areas in a particular urban environment in response to a particular public health crisis. So, if our findings revealed a significant increase in COVID-19 vaccination rates and/or the UI Provision Ratio among the RC-designated census tracts in New Orleans, it would be premature to conclude that the RC model can generalize to other populations and achieve similar results across other cities. To measure such an impact, we would need to conduct a larger study across multiple urban settings. Moreover, we will not use any findings to gauge the impact of the RC model in other types of emergencies, like natural disasters. To do so, we would also need to conduct a larger study, testing the RC model in other contexts, like after a hurricane.

All that being said, three key components of this study – the strength of the secondary measures (and lack of concern around response rates), the relatively large sample (60 total census tracts), and the relevant criteria in building the treatment and comparison census tract groups – all help make this study considerably generalizable to the target population.

3/3 Sample

The study’s target population is New Orleans residents in the areas hardest hit by the COVID-19 pandemic, in terms of both high infections and unemployment rates. There are 184 total census tracts in New Orleans. Given RF’s partnerships with city and state authorities, and the availability of secondary data, the sampling frame of tracts was vast. However, given RF’s staffing limitations and resource constraints, the selected sample consists of the specific census tracts the organization identified to receive the RC. In preparing to deploy the RC members to certain areas, we expect that RF will rank these 184 tracts in terms of COVID-19 impacts (high positivity rates and increases in unemployment since the pandemic began), from hardest hit to least affected. RF then will target the 30 hardest-hit census tract areas with the intervention.

To conduct this evaluation and assess the impact of the RC, we will build a comparison group of 30 census tracts by matching the RC-designated census tracts with other census tracts in New Orleans, based on the two key characteristics the RC used to build its program group:

COVID-19 positivity rates above a certain threshold (determined by RF)

Unemployment rates above a certain threshold (determined by RF)

We will further match the program and comparison groups, to the greatest extent possible, based on two additional traits:

Demographic criteria (racial composition)

Socio-economic criteria (median annual household income)

If more than 30 other census tracts meet ths criteria, then we will randomly select 30 to feature in the comparison group to minimize selection bias.

With a total sample size of 60 census tracts, we will feel confident in the statistical power associated with our evaluation and findings. Based on the way we have assigned program and comparison groups, we think the criteria is inclusive enough such that the sample will be an adequate representation of areas eligible to receive the RC’s services.

Because we will analyze secondary data, we do not have to worry about recruitment for either the treatment or the comparison sample. Similarly, we are not requiring any type of response in our data collection regime and therefore expect 100% follow-up rates, assuming the instrumentation remains stable throughout the course of our evaluation. As a result, we are not concerned about any slippage between the selected sample, the final baseline sample, and the final study sample.

4.5/5 Measures

In order to ensure we can attribute the outcomes of our evaluation to the causal mechanisms of the RC, we must rule out poor measures as a potential confounding variable. It is thus crucially important to use valid and reliable measures. Fortunately, reputable institutions provide the data utilized to measure our two operational definitions in our research questions – vaccine rates and the provision of unemployment benefits.

First-Dose Vaccination Rates

Our first research question seeks to understand the impact that the RC has on first-dose COVID-19 vaccination rates among targeted residents. We will rely on secondary data provided by the Louisiana Department of Health (LDH), the state’s public health agency, which tracks first-dose vaccine rates at the census tract level and issues updates twice per week.

We contend that the LDH’s measure of vaccination rates is a valid and reliable measure in and of itself, given its objective nature and validation by the CDC, which uses LDH’s database to aggregate nationwide COVID-19 vaccine data (“Covid Data Tracker”). Because each observation from the LDH is a ratio data point that can be easily analyzed, we will use the existing measure as provided.

Provision of Unemployment Benefits

Our second research question seeks to understand the RC’s impact on access to UI. The operational definition of this support is the provision of unemployment benefits. We will measure this by calculating the proportion of individuals receiving UI to the total population of unemployed individuals at the census tract level. Using a proportional metric will help control for secular economic trends – such as rapidly changing outlooks across industries due to COVID-19 – that affect unemployment spells. For example, if the RC is successful in assisting unemployed residents in a particular census tract in submitting applications for UI, then we might expect the ratio of unemployed people who actually receive unemployment benefits in that same tract to be higher in the post-intervention period than it was in the pre-intervention periods, even if the overall rate of unemployment declines during that same period.

We will draw the secondary data for the UI Provision Ratio from the Louisiana Workforce Commission (LWC), the organization that administers the state’s UI program. Even though the LWC provides weekly updates on state level UI claims, we expect to have access to such data at a census tract level, given the close partnership between the City of New Orleans and the RC. Due to privacy concerns, the data will likely not include any identifying information.

Much like our measure of vaccination rates, we argue that our constructed measure, sourced from data made available by the LWC, is valid and reliable. It is not a subjective assessment, but rather the proportion between two ratio-level data points provided by the LWC – the total number of individuals receiving UI and the total number of unemployed residents. It is also important to note that the Pew Research Center uses this same metric, albeit pulling data from the Labor Department’s Employment & Training Administration and the Bureau of Labor Statistics (Desilver, 2020).

2/2 Procedures

RF decided to pursue this pilot program in June 2020, which is when this evaluation team formally starts engagement. The full evaluation will take place over the course of 1.5 years, from the process evaluation, through the impact evaluation and the post-impact statistical analyses. In the four months from June 2020 through September 2020, RF will pinpoint 30 New Orleans census tracts to receive the intervention and trained corps members to deploy into these areas hardest hit by COVID-19. The timeline below incorporates those assumptions and outlines our procedures from June 2020 through December 2021.

However, before detailing the full timeline, it is important to note a few points. First, we will not recruit and train an evaluation team, because we are using secondary data and can handle the analysis on our own. Second, while one step of the process evaluation – a comprehensive review of planning documents – will take place prior to launching our outcome evaluation, we will conduct other components of the process evaluation periodically throughout the course of the 1.5 years. Those include regularly reviewing RF’s meeting minutes and progress reports around the RC, as well as conducting site visits (in this case, shadowing the RC members and units in the program census tracts) and key informant interviews (debriefing with the RC members and coordinators on a regular basis). In addition to these routine process-evaluation-related check-ins, we will regularly meet with officials from the New Orleans Health Department to gain insights on any outside effects that could skew the RC’s results.

June 1, 2020 – October 1, 2020:

Assemble list of 30 census tracts for the comparison group based on the 30 RF selected to receive the intervention.

Leverage partnership with the City of New Orleans to gain access to the LWC’s database on unemployment figures.

Because the LWC only publishes state level UI claims data, we will need to access such data at a census tract level.

Scan eight months of data for the LWC’s unemployment and UI measures to assess reliability and validity.

October 1, 2020 – May 1, 2021

Oct. 1, 2020 – Dec. 1, 2021: Step 1 of Process Evaluation

Perform comprehensive literature review of community-based resilience programs.

Complete comprehensive review of RF’s planning documents for the RC.

Oct. 1, 2020 – May 1, 2021: Study Period for 1st Measure (UI Provision Ratio)

For the UI Provision Ratio measure, conduct baseline analyses for the 32 data points in the pre-test period (32 weeks from March 1, 2020 – October 1, 2020). We will first calculate the individual rates for each of the 60 census tracts (30 in the program group and 30 in the comparison group), and then calculate the mean scores for both groups.

Follow up with the LWC staff to ensure there are no substantial changes either the way that unemployed residents are calculated, or the percentage of residents receiving UI. ​

Follow up with RF program staff weekly to ensure there are no substantial changes in programming that could impact the evaluation.

May 1, 2021 – October 1, 2021: Study Period for 2nd Measure (First-Dose Vaccination Rates)

Start to obtain biweekly data from LDH on vaccination rates across the 60 census tracts.

Check in on a weekly basis with LDH to ensure that there are no substantial changes in the way these figures are calculated.

November 1, 2021 – December 1, 2021:

Conduct analyses on the two relevant measures.

For the vaccine measure (pre- and post-test), calculate the vaccination rates for each census tract in the treatment and comparison group, and then calculate the mean scores for both groups. For more guidance, follow the analysis plan below.

For the UI Provision Ratio (pre- and post-test), compare changes in the ratio, across the treatment and comparison census tracts, from the 32 pre-intervention periods to the results from the 32 post-intervention periods. Determine if any results are statistically significant. For more guidance, follow the analysis plan below.

Synthesize and summarize findings from process evaluation and produce a final report.

Synthesize and summarize findings from impact evaluation and produce a final report.

1/1 Analysis Plan

We will begin our analysis of our first research question by calculating the average vaccination rates across the census tracts in the treatment group and in the comparison group at baseline and at our follow-up date. Then, as is customary in this type of 2x2 mixed design, we will calculate the difference in differences between the mean vaccination rates at the pre- and post-test dates for both groups (see Figure B in the Appendix). This will represent the observed program effect of the RC intervention on first-dose vaccination rates. To bolster this analysis, we will conduct a Chi-Square test of independence to gauge if the vaccination rates in the treatment and comparison groups at the follow-up date are likely to be related or not. If the test yields a p-value less than 0.05, we can claim there is a statistically significant difference between both ratios at the five percent level, supporting our original hypothesis.

To analyze the second research question, which concerns the UI Provision Ratio measure, we will use an interrupted time series model. Given the LWC publishes weekly data on unemployment statistics across Louisiana, and we can gather data from before the RC begins in October 2020, we will map out at least 64 data points (the 32 weeks prior to the intervention, and the 32 weeks after). This will help us provide a robust demonstration of the RC intervention’s effects on the UI Provision Ratio in the treatment and comparison groups (see Figure C in Appendix). We decided to include 32 data points on either side of the intervention’s start date due to the nature of UI in Louisiana, in which the maximum time someone can remain on state unemployment benefits is 26 weeks. By providing an extra six weeks in the pre- and post-intervention stages, we can assure the capture of at least one full 26-week cycle.

After inputting all of the observations at the end of the 64th week, we will produce the lines of best fit (or trendlines) for both the treatment and control groups in the period following the RC intervention (Weeks 33-64). We will then analyze any difference in the slopes of the lines. If the treatment group’s slope is more positive or less negative than the slope of the line for the comparison group, then it will suggest the RC intervention led to a greater increase in the share of unemployed residents earning UI – corroborating our hypothesis.

As a final note in this analysis plan, it is important to highlight that this evaluation synthesis will pursue an intent–to–treat (ITT) approach. In other words, we will examine how offering the RC in certain census tracts of New Orleans impacts vaccination rates and the provision of UI. Given that attrition is not a significant concern in this evaluation, we will not pursue a treatment–on–treated (TOT) analysis (which would reflect the effects of actual participation in the program, in the event we did see slippage between those assigned to the treatment and control group, and those who actually participated in both groups).

2/2 IV. Conclusion

The RC pilot program in New Orleans presents a model for cities to help local communities increase their adaptive capacity in the face of disasters. In this proposal, we have outlined an outcome evaluation to test two principal goals of this innovative community resilience initiative: increasing vaccination rates and expanding public awareness (and utilization) of social services. We have proposed a quasi-experimental design, using a program group of the RC-designated census tracts and comparison group of non-RC-designated census tracts, to assess two key research questions related to vaccine rates and UI provisions.

The findings of this evaluation have implications for a range of stakeholders. For one, the study will either bolster or undermine RF’s theory of change. If the evaluation results show that the RC-designated census tracts experienced larger, statistically significant increases in their vaccination rates and share of unemployed residents receiving UI, compared to non-designated tracts, this would support the notion that the RC model boosts local knowledge of health and social services in the face of public health emergencies. RF could then feel more confident pursuing this model in other cities, and it could build support for place-based workforce development programs more broadly. Additionally, it would be of significant value to local and state government agencies that share in the struggle to gain the public’s trust and connect them to supportive services, be it UI or otherwise. On the flip side, if the evaluation produced negative findings, it would push RF to re-assess some of the core assumptions built into their theory of change and tamper enthusiasm around community resilience initiatives across the country. It would also influence whether RF’s funding partners continue working with RF in the same capacity, and inform their criteria for similar funding applicants in the future. While the benefits of targeted, community-based interventions are fairly widely known, there is little research assessing models that specifically enlist residents of a community as public aides in support of their own neighbors. This evaluation will fill that void.

While the results of this study could inform policymakers, government officials, and non-profit leaders in cities across the U.S., there are some notable limitations we must address. First, the evaluation doesn’t fully control for selection bias and contamination – two key threats to internal validity. This is in part because an experimental design using random assignment was not possible in this evaluation due to ethical and practical constraints. That being said, we have mitigated selection bias as much as possible through robust criteria to build a comparison group.

In terms of the risks associated with contamination, it is difficult to conceptualize and identify census tracts that are socioeconomically and demographically similar, operate under sufficiently similar governmental and institutional infrastructures, but are not geographically close. Additionally, the RC model is highly localized, and the environment from one location to the next can be vastly different and threaten comparability between treatment and comparison groups. For example, because UI is administered at the state level, the programs vary vastly across states. Thus factors that either obstruct or facilitate UI applications and the provision of benefits may vary between Louisiana and New York, for example. Moreover, the fluidity of the geographical boundaries of neighborly exchanges makes it difficult to isolate primary and secondary effects of the program. In other words, it is difficult to determine which behavior changes stem directly from the program’s corps members, and what changes were influenced by other civilian neighbors.

Overall, while RF, New Orleans government officials, and city residents will likely find this outcome evaluation valuable, the findings are not applicable on a wider scale to other cities and other disasters – public health or otherwise. To more broadly understand the impact of innovative community resilience models nationwide and across other emergencies, evaluators would propose a large-scale evaluation that includes other cities, especially those with different public health administration and social service infrastructures (examples include different regulations for UI and different hierarchies of authority across agencies tasked with administering services). Future evaluations of similar programs should also more thoughtfully wrestle with another key variable: racial composition. New Orleans is a predominantly Black city, and research shows that different ethnic groups can have complex and varying feelings toward government authorities (Best et al., 2021). As such, the impact on trust in public officials, especially in health administration, may vary in places with other racial groups in the majority.

Writing, style & organization: 2/2 Very well written (and very little passive voice!)

29.5/30 =98%

Outstanding proposal! You worked hard on this one and it really paid off.

Appendix

Figure A – Logic Model

Figure B – Example Analysis for Research Question #1

Example Analysis Comparing Vaccine Uptake Across Treatment and Comparison Groups

Figure C – Example Analysis for Research Question #2

Example Interrupted Time Series Model

References and Works Cited

Barton, A. H. (1970). Communities in Disaster: A Sociological Analysis of Collective Stress

Situations. Ward Lock Educational.

Best, A. L., Fletcher, F. E., Kadono, M., & Warren, R. C. (2021, March 24). Institutional Distrust

among African Americans and Building Trustworthiness in the COVID-19 Response:

Implications for Ethical Public Health Practice. Journal of Health Care for the Poor and Underserved, (32)1, 90-98. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7988507/.

Centers for Disease Control and Prevention. COVID Data Tracker. Retrieved from:

https://covid.cdc.gov/covid-data-tracker/#vaccinations_vacc-people-onedose-pop-5yr.

Centers for Disease Control and Prevention. (2020, December 10). COVID-19 Racial and Ethnic

Disparities. CDC. Retrieved from:

https://www.cdc.gov/coronavirus/2019-ncov/community/health-equity/racial-ethnic-disparities/index.html.

Centers for Disease Control and Prevention. “COVID-19 Vaccination Data

Systems and Data Sources.” Retrieved May 15, 2022, from:

https://www.cdc.gov/coronavirus/2019-ncov/vaccines/data-systems-sources.html.

Clendaniel, M. (2021, February 18). How the New Orleans Resilience Corps Helped the City

Fight the Pandemic. Fast Company. Retrieved from:

https://www.fastcompany.com/90606116/how-the-new-orleans-resilience-corps-helped-the-city-fight-the-pandemic.

Desilver, D. (2020, April 24). Not All Unemployed People Get Unemployment Benefits; in

Some States, Very Few Do. Pew Research Center. Retrieved from:

https://www.pewresearch.org/fact-tank/2020/04/24/not-all-unemployed-people-get-unemployment-benefits-in-some-states-very-few-do/.

Devitt, J. (2018, January 29). Why Do We Trust, or Not Trust, Strangers? The Answer is

Pavlovian, New Psychology Research Finds. NYU. Retrieved from:

https://www.nyu.edu/about/news-publications/news/2018/january/why-do-we-trust--or-not-trust--strangers--the-answer-is-pavlovia.html.

Funes, Y. (2021, March 15). In NoLa, a Resilience Corps for Tomorrow’s Disasters. Atmos.

Retrieved from: https://atmos.earth/new-orleans-resilience-corps-covid.

Klassen, T. P., MacKay, J. M., Moher, D., Walker, A., & Jones, A. L. (2000). Community-Based

Injury Prevention Interventions. The Future of Children, 10(1), 83–110.

Retrieved from: https://doi.org/10.2307/1602826.

Louisiana Dept. of Health. COVID-19 Information. Retrieved April 20, 2022, from:

https://ldh.la.gov/Coronavirus/.

Mao, G., Drury, J., Fernandes-Jesus, M., & Ntontis, E. How participation in Covid-19 mutual aid

groups affects subjective well-being and how political identity moderates these effects.

Anal Soc Issues Public Policy. 2021; 21: 1082– 1112. Retrieved from: https://doi.org/10.1111/asap.12275.

Nugent, Ciara (2021, September 10). Cities Have Firefighters and Trash Collectors. As the

Climate Breaks Down, Do They Also Need Resilience Corps? TIME. Retrieved from:

https://time.com/6096397/climate-change-resilience-corps/.

Rashford, M. (2021, March 22). Black Residents in New Orleans Hit Hardest by Covid. NBC

Nightly News. Retrieved from:

https://www.nbcnews.com/nightly-news/video/black-residents-in-new-orleans-hit-hardest-by-covid-108941893573.

Resilience Corps. Resilience Force. Retrieved from: https://resilienceforce.org/resilience-corps/.

Resilience Force (2020, April 2). Resilience Corps White Paper. Retrieved from:

https://resilienceforce.org/wp-content/uploads/2020/04/1-coronavirus-resilience-corps-lt.

pdf.

Rockefeller Foundation. (2021, May 7). A Covid-19 Resilience Team Prepares for the Next

Disaster. Rockefeller Foundation. Retrieved from:

https://www.rockefellerfoundation.org/case-study/a-covid-19-resilience-team-prepares-for-the-next-disaster/.

Schwartz, R.M., Gillezeau, C.N., Liu, B., Lieberman-Cribbin, W. & Taioli, E. (2017).

Longitudinal Impact on Hurricane Sandy Exposure on Mental Health Symptoms.

International Journal of Environmental Research and Public Health, 14(9), 957.

Simmons, S. (2021, May 13). CDC And FDA Trusted By Only Half Of Americans. NPR.

Retrieved from: https://www.npr.org/2021/05/13/996331692/poll-finds-public-health-has-a-trust-problem.

Tierney, K. (2014). The Social Roots of Risk: Producing Disasters, Promoting Resilience.

Stanford University Press.

U.S. Census Bureau. Louisiana - 2020 Census Tracts. Retrieved from:

https://www2.census.gov/geo/maps/DC2020/PL20/st22_la/censustract_maps/.

Ward, P. (2017, October 12). Understanding the perceived logic of care by vaccine-hesitant and

vaccine-refusing parents: A qualitative study in Australia. PLOS. Retrieved from:

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0185955.

Williams, J. (2020, July 2). For Black New Orleans residents, coronavirus death toll more

alarming than previously thought. NOLA.com. Retrieved from: https://www.nola.com/news/coronavirus/article_170f9226-bbbf-11ea-b194-27f84d860372.html.

Woodruff, E. (2021, March 22). The City's Strategy for Vaccinating New Orleans' Most

Vulnerable? Knocking on Their Doors. NOLA.com. Retrieved from:

https://www.nola.com/news/coronavirus/article_1d5fd4f6-8b5d-11eb-b733-13dbdaf447f6.html.