Evaluating Programs and Policies

Outcome Evaluation Proposal for

The Youth Homelessness Demonstration Program

2/2 Specific Aims

In response to lack of affordable housing and rising eviction rates in the greater Houston region, The Coalition for the Homeless of Houston/Harris County established a local homelessness response system called The Way Home. The Way Home aims to improve system coordination among housing service providers and to ensure easy access to resources for homeless youth in Pasadena, Conroe, Harris, Fort Bend, and Montgomery counties. The Way Home is launching a new program called The Youth Homelessness Demonstration Program (YHDP), an initiative The Way Home designed to reduce the incidence of Youth and Young Adult (YYA) homelessness through a “coordinated community approach.” Since this is a new program, no one has done an outcome evaluation for it before. Our evaluation of YHDP has the potential to contribute to critical decisions that The Way Home stakeholders make in the future. Program improvements that result from these decisions are ultimately most beneficial to the homeless YYA and local community. The evaluation will also help us determine if more funding should be invested in the program, and if the YHDP model should be replicated in other cities.

We will conduct a quasi-experimental design study that compares outcomes of YYA with access to YHDP to a contemporaneous comparison group of YYA who are not exposed to the program. We will select the comparison group from five other comparable counties in Houston. We will further establish comparability between our groups by collecting and matching data on participants' race, gender, age, socioeconomic status, total time homeless, and criminal background. Our proposed outcome evaluation will focus on YHDP’s program goal to “identify all youth and young adults experiencing homelessness as quickly as possible and connect them to permanent housing options and other non-housing resources.” In this proposal, we will assess the stated program goal through the following two research questions:

Are homeless YYA more likely to receive legal services for housing applications within one year of being identified by YHDP than homeless YYA not exposed to the program?

Within a year of identification as homeless by YHDP outreach teams, do YYA who are identified as homeless by YHDP move into permanent housing more quickly, compared to homeless YYA who do not interact with the program?

4/4 Background & Significance

Background

Between 2000 and 2016 the Houston population grew by approximately 15% while the median sale price for a home increased by approximately 20%. In addition, a growing number of renters are having difficulty paying rent and as of 2019, one in 25 renters were evicted from their homes. These trends contribute to increases in YYA homelessness as it has become too overcrowded and too costly to live in this area. In 2022, the Greater Houston Community Foundation reported an increase in the unsheltered homeless population specifically in Harris, Fort Bend, and Montgomery counties of Houston. They also found that 2022 has been the first year that the unsheltered homeless population has outnumbered the sheltered homeless population.6 Among groups that remain at greater risk of experiencing homelessness compared to other subpopulations, unaccompanied youth (under age 18) and young adults (ages 18-24) who are without a parent or guardian have experienced higher incidence of homelessness during this period.6

YYA experience homelessness because of “interconnected challenges” that need system-level and youth-level solutions. These challenges include housing policies that disqualify YYA with criminal records from obtaining leases or that “prevent families from accepting justice-involved youth back into the home.” Such YYA often require access to legal services for record sealing or expungement to qualify for lease-based housing.

In addition to the negative impacts on the safety and stability of youth and young adults, homelessness creates a financial burden on the community. Non-permanent housing options such as emergency shelters are critical but incredibly costly interventions. Local communities also spend a significant portion of taxpayer dollars on homelessness related to “emergency services, police intervention, incarceration, and street cleaning.” The problem of YYA homelessness therefore poses a significant challenge not only for young people themselves, but also for our local governments and communities at large.

YHDP seeks to reduce the incidence of YYA homelessness through a coordinated community approach. This program is led by The Way Home, an umbrella organization of nonprofit organizations, community stakeholders, homeless service agencies, local governments, Public Housing Authorities and the Michael E. DeBakey Department of Veteran Affairs Medical Center. Together, these organizations seek to end and prevent homelessness in the greater Houston region. After seeing promising results from the organization’s approach, they have decided to turn to the urgent issue of youth homelessness. The program focuses on (1) unaccompanied minors and (2) young adults ages 18-24 who are not accompanied by a parent/guardian. It serves youth across the following counties in Texas: Houston, Pasadena, Conroe/Harris, Fort Bend, and Montgomery.

YHDP defines homelessness as living “in a place not meant for human habitation (including the streets or in a car), emergency shelter, transitional housing, and/or hotels paid for by a government or charitable organization.” YHDP is built on the belief that by providing a variety of supportive services (known as the Continuum of Care) to homeless YYA, the program can ensure such youth find stable, safe housing – in other words, that youth can find a place they can call home. YHDP utilizes a “youth-centered approach emphasizing youth choice for both housing and supportive services that can meet their individualized needs.” Ultimately, the program seeks to end homelessness among this population – with any homelessness among YYA that persists in these counties being rare, short, and non-recurring. The program believes that in turn, through finding stable housing, young people are empowered for their futures – especially since lack of stable housing can be a significant blocker in connecting to educational and professional opportunities.

The program therefore works to proactively identify homeless YYA, to offer them services such as legal support, trauma-informed health care, and employment assistance, and to ultimately connect them to permanent housing. The program has YHDP outreach teams who identify homeless YYA through street outreach, working with schools, and working with youth-serving organizations to engage with these young people. Once homeless YYA have been identified, the program’s system navigators help young people access numerous youth services, each offered by different government agencies and youth service organizations. This includes legal assistance on housing applications in order to reduce legal barriers to stable housing, especially for young people who have been involved in the Juvenile Justice System. In order to ensure the program has access to adequate youth services and supportive housing options, it recruits youth service providers, as well as landlords who provide different types of housing, from emergency housing to more long-term, affordable housing. Overall, the program’s coalition of community partners seeks to provide a more streamlined, easy-to-navigate experience for YYA, allowing YHDP to identify more homeless young people, connect them to the needed services, and place them into more permanent housing as quickly as possible.

Significance:

As described above, The Way Home’s overall program goal is to prevent and end youth homelessness through its Continuum of Care. This is an emerging model with many policy implications; if found to be effective at achieving its program goal of connecting homeless YYA to services and permanent housing, cities across the country could benefit from replicating its success. To assess whether the program is indeed effective, The Way Home has implemented an internal system to measure success, called a Continuous Quality Improvement (CQI) process that measures intervention results. With regards to stable housing, program staff conduct the CQI process by looking, on an ongoing basis, at the number of homeless YYA identified and the “utilization rates of housing and service pathways within the YHDP homeless response system.” Since YHDP is a new program, the CQI evaluation results are not yet available. While the CQI is a valuable internal tool, it is not sufficient to evaluate the impact of this program. Our outcome evaluation will provide an unbiased assessment of the program intervention that is independent of the CQI process. Our evaluation will not be directed by program staff themselves, so we will eliminate potential bias of program staff who want to demonstrate success. The CQI data also lacks a comparison group, which weakens its internal validity; it will not be possible to tell from the CQI data if the results are attributable to the program itself. Lastly, our evaluation includes a process evaluation, which makes it more scientifically rigorous than the CQI.

YHDP’s program goal aligns with that of the U.S. Department of Housing and Urban Development (HUD) Continuum of Care Program. HUD’s CoC “promotes community-wide commitment to the goal of ending homelessness and emphasizes identifying needs and building a system of housing and services to address those needs.” Although there is literature on HUD’s CoC program and other housing assistance programs, there is a lack of research on youth-centered programs that emphasize the need for permanent housing. Existing studies also tend to focus on specific subpopulations of homeless YYA, such as youth with HIV or mental illness, and interventions that enhance access to healthcare rather than to permanent housing. Our evaluation is necessary to not only assess the efficacy of this specific program in Texas, but also to gather insights that apply more broadly to youth-centered programs that aim to increase permanent housing connections and encompass a broad range of homeless YYA subpopulations.

While YHDP has an innovative, collaborative model for working to end homelessness among YYA, our evaluation is needed in order to assess whether the program is truly more effective than the status quo. HUD has invested significant funds in this program (over $10.3M), and is deciding whether to invest more in YHDP, both for Houston as well as for other cities across the U.S., as youth homelessness is a significant issue across many major metropolitan areas. However, before we can scale the program and receive more government funding and resources, we need to ensure the program actually makes a positive difference for homeless YYA in the counties in which it is implemented. An evaluation that assesses the program’s efficacy across two key measures will help prove – or disprove – that this program actually works. Our two key research questions are:

1. Are homeless YYA more likely to receive legal services for housing applications within one year of being identified by YHDP than homeless YYA not exposed to the program?

2. Within a year of identification as homeless by YHDP outreach teams, do YYA who are identified as homeless by YHDP move into permanent housing more quickly, compared to homeless YYA who do not interact with YHDP?

These two research questions speak directly to the program’s theory and goals: we will evaluate the efficacy of the program both in connecting youth identified as homeless with more services that will help them access stable housing, as well as in actually placing homeless YYA into permanent housing. We define permanent housing as a housing situation that is meant to be permanent, such as living with family, living with friends, or renting an apartment.

Research Design and Methods

3/3 Process Evaluation

The goal for our evaluation is to assess the effectiveness of the YHDP at accomplishing its goal of “identifying YYA experiencing homelessness and connecting them to available resources.” In particular, as described above, we are looking at whether YYA who the YHDP outreach teams identify as homeless are more likely to receive legal support (legal resources) and to more quickly connect to permanent housing than those who do not have access to the YHDP programs. In other words, the two outcomes we are looking at with our evaluation are connection to legal services, and placement in permanent housing. At the end of our outcome evaluation, if we see positive results, we would want to understand what actually happened to contribute to those results. In addition, we want to ensure that if we do not see an impact to legal support and permanent housing at the end of our evaluation, we can confidently say that the lack of efficacy of the program itself is to blame, and not flawed implementation or flawed process assumptions. In other words, we want to be able to answer the questions: (1) What does YHDP actually look like in the real world? and (2) Was YHDP rolled out effectively and in alignment with our assumptions? Before we jump into evaluating the outcomes, we first need to look at the activities and outputs – and see if the program is being implemented as planned across these areas.

YHDP is fairly broad with many elements to it, so for our process evaluation we will focus just on the program’s activities that are most connected to our two research questions. This includes looking at activities that are tied to connecting homeless YYA with legal services, as well as those tied to connecting YYA to permanent housing. With this in mind, we would carry out the following activities for our process evaluation:

First, we want to ensure that the YHDP outreach teams are effectively identifying homeless YYA, and engaging them in the intake process to participate in the program. In order to do so, we will observe the activities of the outreach teams as they engage homeless YYA they find on the street, in schools, and through other community partners. We will observe and document how they identify homeless YYA, how they speak with them, and how they encourage them to work with YHDP to connect to housing.

From there, we will confirm that a substantial portion of the young people who go through the intake process with the YHDP outreach teams do end up participating in the YHDP programs. If not, that would be a flaw in our process assumptions; if young people are going through the intake process but then are not participating in the activities offered to them through YHDP, it would undermine the belief that the program is actually taking YYA from identification through to connection to resources. We will do this through document review – reviewing the reporting generated as YYA go through the program to ensure young people are not dropping out of the program after intake.

We will also make sure that there are adequate legal services and permanent housing options available for the homeless YYA who are identified by the YHDP outreach team – and that the programmatic staff is adequately trained and staffed in order to make the connection between these young people and these resources. We will ensure this through a combination of document review and key informant interviews/focus groups. In terms of document review, we will review the housing and legal services available to YHDP to ensure that there are truly enough for all of the young people identified; if not, that would be an issue with the assumption that the services we want to offer these young people are actually available to them through YHDP. In terms of interviews and focus groups, we will speak with both homeless YYA who are participating in the program, and staff who are tasked with connecting these YYA to legal services and helping them get into permanent housing. We want to understand the specific steps that these staff members are taking to support the homeless YYA, whether the YYA are receptive, what challenges the staff and YYA are facing, and what is going well.

6/6 Design

For our research design, we will conduct a quasi-experiment and use matching to establish a contemporaneous comparison group. Our treatment group consists of YYA that the YHDP outreach team identifies as homeless in Pasadena, Conroe, Harris, Fort Bend, and Montgomery counties in Texas. Our comparison group consists of YYA from five other counties in Texas who have specifically stated that they would be interested in a program like this (but no such program currently exists in those counties) and who we will select based on comparability at baseline in terms of demographics such as race, gender, age, and socioeconomic background. In addition, there are many factors that contribute to YYA homelessness that need to be considered when selecting a viable comparison group. We are also looking for comparability in terms of variables related to our outcome variables (factors that put youth at risk of becoming homeless) such as total time homeless, family conflict, “aging out” of foster care or juvenile justice systems, criminal history and sexual orientation.

We have selected a quasi-experiment, rather than a randomized controlled trial (RCT), because this is a county-wide program. All YYA in the treatment counties have access to the resources of the program and it would be nearly impossible to provide the program to some individuals and not others within a county making this difficult to randomize at the individual level. For the treatment counties, the program has already been rolled out and so that also poses a problem of randomizing some YYA to a treatment and control group. We expect many YYA to have participated in the program in some capacity as it is an already established program within the five counties in Texas. In addition, given the nature of this program, contamination would be an issue if we did a RCT because there is no way of controlling homeless YYA from sharing information and resources with each other. It would also be ethically wrong to deny YYA within our treatment counties services and resources if they are in need just for the sake of this evaluation. All things considered, our best option is to select a group of YYA who have been exposed to the program across all treatment counties and select comparison counties in other areas of Texas. The comparison counties that will not receive the intervention will act as a counterfactual and tell us what potentially would have happened in the absence of the intervention. A quasi-experimental design is therefore ideal in this scenario because randomization is not feasible and this type of design will give us a good estimate of the impact of the intervention, as long as we find a viable comparison group.

In order to find such a viable comparison group, we will engage in an extensive matching process between our program group and our comparison group. We will begin by matching at the county level by aggregating baseline characteristics using census data (i.e., choosing counties as a whole who are comparable to the treatment counties). We will choose the counties that are most similar to our treatment group counties with respect to demographics and observed characteristics that put youth at risk for homelessness. The more similar these counties are, the better they will act as a comparison group for our group of YYA that have participated in YHDP. As described in our “Sample” section below, we will then further strengthen our comparison group by ensuring that the final samples themselves are also comparable at baseline across demographic and other relevant characteristics. Overall, we will match based on aggregated data rather than on a one-to-one basis because one-to-one matching would be overly cumbersome and would lower our sample, and therefore our statistical power.

There are many threats to internal validity that are apparent with this type of experimental design, but we will take action to decrease their threat. The key threats in question include attrition, testing effects, maturation, outside effects and selection bias. We are relying heavily on primary data from surveys (from both the program group and the comparison group) and so, attrition is a problem if we receive low response rates. Testing effects arise in both the treatment and comparison group if YYA behave differently due to the pre-test itself or because they know they are a part of a study. Maturation is an issue here because we are observing youth who are at peak ages of growth and development both mentally and physically. Outside effects are an issue because certain events may occur in one county that do not occur in others that will cause a change in our outcome variables. Finally, selection bias is an issue because we lack random assignment in our experimental design. Alternatively, contamination and compensation are not issues with this quasi-experimental design because it is unlikely that both groups will interact with each other since they live in different counties. We have no reason to believe that the comparison group will behave differently than they would have otherwise because we are not exposing them to the program or giving them any indication that the program exists. Instrumentation will not be an issue as well because we have developed sound measures for our outcome variables.

To combat the threats to internal validity, we will provide incentives for the program group and the comparison group to conduct post-tests and reduce attrition. We will also conduct different forms of communication including email, text messaging and follow up calls to try to improve our response rates. To combat maturation, we will ensure that the ages of YYA in our program group and in our comparison group are comparable, so they would be at a similar phase of development. In addition, we will add shadow controls to our design by interviewing YYA and staff about any outside effects that could have caused the changes in the outcome variables. We will reduce selection bias (however not eliminate) because of the way we match counties between the treatment and comparison groups. We will find a comparison group that is as similar as possible to the treatment group at baseline, to reduce the risk that differences in the groups themselves have led to any observed impact of the intervention. In addition, due to the voluntary nature of the program and the voluntary nature of the evaluation for the comparison group, we can argue that both the treatment and comparison groups are similar in their desire to participate in this type of program – which will give us more confidence that changes in our outcome variables cannot be attributed to things like motivation and desire. Lastly, we do not believe that testing effects or regression to the mean will have an impact on our results because we have no reason to believe that these effects would occur in one group and not the other – so any additional effect we see will likely be attributed to the program.

3/3 Sample

For our intervention group, our target population is all YYA who the YHDP outreach teams identify as homeless and who opt into the program. This is because ultimately this is a voluntary program; not all homeless YYA who are identified by the YHDP outreach teams will opt into engaging with the program, and our goal is to assess outcomes for only those who do opt in through the intake process. Our sampling frame is all the YYA who provide the necessary information about themselves to the outreach team through the intake process. The data collected by the YHDP outreach team will serve as our pre-test for the treatment group and as the basis for ensuring comparability of our comparison group. The data collected will include demographic information, information about how long they have been homeless, criminal history, and any other factors we determine are relevant for matching. Because we will incorporate our pre-test into the intake process by the program’s existing outreach team, we expect the sampling frame to closely resemble the target population.

That being said, while everyone for whom we have data from their intake process could be part of our sample, ultimately our final sample will only include data from those individuals for whom we will also receive data for our post-test. YHDP serves around ~700 youth annually who are identified by YHDP outreach teams. Given that we will collect contact information from all participants, and that we will track their progress through the program as they work with our program staff and service providers, we expect that reaching a response rate of 70-75% on our post-test is extremely reasonable. This would leave us with a final sample of ~500 people for our intervention group.

As for our comparison group, we will have evaluation outreach teams go out into our comparison counties and collect the same data that the YHDP outreach teams collect from YYA in treatment counties – such as demographic information, information about how long they have been homeless for, etc. The evaluation outreach team will ask for this information by explaining that they are part of a study on how to better support homeless YYA; instead of offering a program to go through, the outreach teams will ask the homeless YYA whether, if given the opportunity to engage in programs to find shelter and/or permanent housing, they would want to do so. If the YYA says yes, the evaluation outreach team member will continue on to the additional questions; if not, the evaluation outreach team will move on. This will allow the outreach teams to get the information they need for the pre-test and to establish comparability of the comparison group to our intervention group. Ultimately, we want to compare intervention group members who opt into the program with comparison group members who would likely opt into the program, and if people say they wouldn’t be interested in a program to find shelter/permanent housing, they probably would not opt into the program in their county and therefore shouldn’t be a part of our sample.

Our goal is to collect pre-test data from 1,000 homeless YYA in our comparison counties. This is because we want to have more flexibility with this group given we will likely have a lower rate due to a lack of sustained interaction with them throughout the year. Ultimately, our goal is to have at least 700 people in our comparison group sample who respond to both the pre-test and the post-test. This will provide us with flexibility to maintain high statistical power while adjusting our comparison group sample as needed – both using statistical methods as well as potentially removing some members of our comparison sample to ensure the final comparison group fully matches the intervention group. If we have trouble reaching 1,000 homeless YYA for our comparison counties, we will use snowball sampling in order to reach more homeless YYA, since young people are likely to know each other.

As a quasi-experiment evaluating a real-life policy (without significant intervention from the evaluators), this study will have strong external validity. The sampling frame for our intervention group will closely mirror the target population, and our goal is to use as much of the sampling frame as possible in the sample. Therefore, as long as we get a 70%+ response rate on our post-test from both intervention group and comparison group, our study will have strong external validity – since there won’t be a large leap from the final sample to the target population. In addition, regardless of how much engagement YYA that are identified by the YHDP outreach team actually have with the YHDP programs throughout the year, we will still include them in our post-test and evaluation. This reflects an “Intent to Treat” approach; ultimately our two measures of success rely on the ability to funnel YYA who are identified as homeless through the YHDP programs, and into positive outcomes such as receiving legal assistance and finding permanent housing. If youth are identified as homeless by the YHDP outreach teams but then do not actually receive much support from the program, that should still be reflected in our analysis of the program’s efficacy. Our “Intent to Treat” approach will also strengthen our external validity as we look realistically at what happens in a county where youth are given the opportunity to participate in a program like YHDP – whether or not they fully take advantage of this opportunity.

5/5 Measures

We will incorporate both of our measures into our pre-test and post-test surveys for each participant. We will enroll participants with our pre-test on a rolling basis until our target group sizes are met, and we will administer our post-test survey one year after baseline. The baseline period starts at the intake date of the first identified program participant. The one-year period ends 12 months after the intake date of the last identified program participant. We will distribute the post-test surveys by email and phone calls using contact information collected by the YHDP and evaluation outreach teams during the pre-test.

To determine the number of the homeless YYA receiving legal services, we will use survey responses of YYA self-reporting whether they received legal assistance. Since this research question does not reflect a pre-test for our outcome variable (legal services), there will not be a question specific to this measure on our pre-test survey. The survey question on our post-test will ask program participants and our comparison group if they received legal assistance for housing applications in the last year. This level of measurement is nominal because we are asking individuals if they received legal assistance for housing applications. A potential limitation of this measure is low response rates from the comparison group. To minimize attrition, we will obtain at least two backup contacts for every participant and conduct numerous follow-ups with non-respondents via emails and phone calls. Participants will also receive an incentive for every survey completion.

This measure has sufficient face validity because we are defining “receiving legal services” as having completed an initial consultation with a lawyer or legal assistant. This is determined objectively. A potential limitation is false positive reports because of participants’ poor recollection or bias to falsely report. To measure test-retest reliability, we will test a sub-sample (N=20) from each group by administering a second survey two weeks after their post-test survey completion. We will ask participants if they received legal assistance in the last year. If this measure is reliable, then responses will remain unchanged after the two-week follow-up. To verify the self-reported data, we will also cross-reference using enrollment data collected by YHDP outreach teams. For program participants, we will also verify their responses through their legal services. We will try to further verify the responses from the comparison group through court records.

For the second research question, we will measure the number of days between when homeless YYA are identified by outreach teams and when they move into permanent housing. This measure is interval-ratio level data (continuous), since it is measured in number of days. As described above, we will obtain this information via our post-test survey one year after baseline to determine whether YYA were connected to permanent housing since being identified by outreach teams, and if so, when they moved into permanent housing. Where possible, we will verify accuracy of the self-reported data using secondary data. For YYA that are identified as homeless by YHDP, we will have data that indicates when the YDHP outreach teams first interacted with the young person and will also have data on when that person moved into a permanent housing situation – since the outreach teams will collect this data from program participants on an ongoing basis. We will also use data on housing applications, lease records, and move-in dates to cross-reference the self-reported data from participants. For this research question, we will only look at when YYA moved into their permanent housing, not if they subsequently moved out.

To assess test-retest reliability, we will test a sub-sample (N=20) from each group by administering a second survey two weeks after their post-test survey completion. In our post-test survey, we will ask participants in the program and comparison groups who have moved into permanent housing to provide their move-in date. We will ask the sub-sample of participants the same question after two weeks. If this is a reliable measure, then the response for each participant’s move-in date should remain unchanged for each test. Using statistical analysis, we will test for a correlation coefficient of 0.70 or higher for the permanent housing move-in date. To measure reliability of the secondary data that we use to verify survey responses, we will examine YHDP’s data collection procedures. We will determine if data coding and double coding is involved to make distinctions among durations in homelessness or subpopulations of YYA. If coding is involved, we will examine the training procedures of staff and if routine data quality checks are performed.

The number of days before moving into permanent housing is an objective and concrete variable (since it is a number), so the face validity is sufficient. However, the definitions of homelessness and permanent housing themselves are somewhat subjective. Therefore, overall we are also relying on content validity, since we are using the established definitions from experts in the field.

2/2 Procedures

Step 1: This evaluation will start off with an examination of the YHDP’s process/program theory to determine the key assumptions of the program. This is important because it will provide us with a framework of what we need to look out for in our subsequent Process Evaluation. Based on our research of the program thus far, we have determined the following assumptions, but want to dig in deeper as the first step of our outcome evaluation:

We will find providers across the program’s continuum of care and they will have interest in participating in the program.

We will have enough providers to serve the number of YYA experiencing homelessness.

We will have enough landlords willing to provide supportive housing.

There is enough housing available in Houston to provide stable housing to all homeless YYA.

Homeless YYA will want to participate in this program.

Providers become open to sharing data about the YYA they serve.

YYA become open to sharing honest information about their housing situation.

Step 2: In the next step, we will conduct a one-month long process evaluation of the implementation of the program. For the process evaluation, we will focus on how YYA are being identified, the protocol being used during the intake process, and the protocol being used to connect YYA to permanent housing and other services such as legal services. We will assess staff functions, and understanding of the program and its goals.

Step 3: Next, we will leverage county-level demographic data to select our five comparison counties. We will select five comparison counties to match the five intervention counties.

Step 4: We will then create pre-survey questions for the comparison group including screener questions to determine eligibility to be in the sample and follow-up questions for participants that are deemed eligible. These should mirror the pre-survey questions for the intervention group as much as possible.

Step 5: In our five comparison counties, our evaluator outreach team will begin the process of collecting data from homeless YYA they identify in the comparison counties. The evaluation outreach teams will go to the comparison counties to survey potential participants for the comparison group. Evaluation outreach teams will tell potential participants that they are participating in a study that aims to provide information about YYA homelessness and the types of resources needed to end youth homelessness. We will offer participants an incentive to participate and complete the follow-up survey one year later.

Step 6: During the same time frame, the YHDP outreach teams will collect data from homeless YYA for the treatment group as part of the intake process for the YHDP. This data will provide us the sampling frame from the target population for the treatment group. Once we have reached our target size for our treatment group (1,000 pre-tests) and comparison group (700 pre-tests), we will stop collecting and entering pre-test data into our tracking system. Throughout this process, we will work to establish comparability between the comparison group and the treatment group based on the baseline characteristics observed in the treatment group during the pre-test.

Step 7: Once one year has passed since the first pre-test was completed, we will begin to administer our post-test. Our evaluation team will conduct follow-up surveys for the identified YYA to determine the effect of the intervention. As needed, we will engage in various methods of communication (text, email and phone calls) to increase response rates. We will give the post-test survey to participants on a rolling basis so that post-test dates are administered exactly one year after each participant's pre-test date. We will include specific questions on the post-test survey that speak to our two measures: (1) asking whether or not the participant received legal support for housing applications over the last year, and (2) asking the date on which the participant moved into permanent housing (if they do so.)

Step 8: We will then analyze the data using difference-in-difference and cross-sectional estimation of the effect of the intervention over time for our two measures: usage of legal services for housing applications, and number of days between participants being identified as homeless and when they moved into permanent housing.

Step 9: Lastly, we will compile and discuss assumptions, key findings, and limitations with the evaluation and the program itself. Through doing so, our goal is to fully assess and understand the impact of the program and its need in the fight against YYA homelessness.

.75/1 Analysis plan

The YHDP outreach teams (not our evaluation staff) will conduct the pre-tests for the treatment group during their intake process for the program. The evaluator outreach teams will conduct the pre-tests for the comparison group as well as post-tests for both the treatment and comparison groups. During the pre-test process, each outreach team will conduct one-on-one interviews with participants at which time they will input responses into a tracking system. The tracking system will have participant responses as well as personal information, demographic information, and contact information of both the YYA and other people who we will reach in the event that we cannot one year following their pre-test. The system will store the data and be able to produce reports on response rates during the follow up interviews to be able to track who needs more follow up communications. The tracking system will also be our means of knowing when participants have completed a year in the study as we will enroll participants on a rolling basis, giving them different pre-test dates.

For our first research question (about connecting YYA to legal services), we will determine the treatment effect by using a cross-section estimate which measures the difference in the proportion of YYA that received legal services for housing applications in the intervention group to the proportion of YYA that received legal services in the comparison group. For our second research question (about the time it takes for a participant to become permanently housed), we will use a difference-in-difference analysis by comparing the change over time in the number of days it takes for YYA in both the intervention and comparison groups to move into permanent housing. Difference-in-difference models rely on an assumption of less strict exchangeability, meaning in the absence of the intervention, the unobserved differences between the intervention and comparison groups are the same over time. This approach is ideal when randomization at the individual level is not feasible, which is our current case. In addition, this approach removes biases in the post-intervention period between the intervention and comparison groups that could result from permanent differences between the groups.

2/2 Conclusion

YHDP has a lofty goal of ending YYA homelessness and providing YYA with the resources needed to thrive. A robust evaluation is needed to assess the effectiveness of the YHDP and determine if the program is or is not reaching its goals. YYA homelessness has now become a growing problem in many other major cities and an evaluation of this scale will determine if a program like this should be replicated in other places. Our biggest limitation is that this is not a randomized control trial and therefore, we cannot eliminate selection bias. However, we have determined strong solutions to strengthen this quasi-experimental design and reduce the effects of key threats to internal validity. Our plan to combat attrition is to prevent it and so, we will use multiple methods of communication to keep response rates up so that the composition and comparability of our comparison group remains the same. Although selection bias cannot be completely eliminated, we will reduce it significantly through our matching process for the comparison group. We are being strategic in selecting comparable counties using census data and then once our sample is selected, we will establish comparability again based on specific baseline characteristics that are apparent in the treatment group. We will reduce maturation by choosing a sample of YYA that are at similar stages of emotional development and similar age levels. Outside effects are mitigated by our use of shadow controls. Finally, we do not expect the effect of a pre-test or the Hawthorne effect to be large because both groups will likely experience these effects if they do occur. Therefore, even with the limitations of this design, we believe this evaluation is a strong way to assess the effect of the YHDP program intervention. This evaluation is extremely beneficial because it will add to the research surrounding the homeless YYA population, which is an area that is lacking in studies and research. As of right now, we do not know for sure what works, but both a process and outcome evaluation will shed some light on this. Key findings from this evaluation will provide information about the needs of this population and will ultimately help aid in housing youth across the country for generations to come.

Writing, style & organization: 2/2; very well written and proofed, and minimal passive voice

29.75/30 Outstanding proposal! A tiny stumble in the analysis plan, but otherwise virtually flawless.