Research Proposal Option: Proposal to Conduct an Evaluation

(Evaluation Template)

Proposed Outcomes Evaluation of the Maryland Department of Housing and Community

Developments Homelessness Solutions Program

HM893

January 26, 2024

Note: You are provided recommended page lengths per section to guide your writing efforts. Your entire paper should be a maximum of 30 pages (including Title Page, Abstract, References, and Appendices)

Abstract (150 to 250 words maximum): The abstract should contain the following information:

What is the public health problem addressed?

What is the program/policy you plan to evaluate?

Who is the target population?

What is/are the main evaluation question(s)?

What is the evaluation design?

What are the expected implications of the evaluation?

What data will be collected and used to conduct the evaluation?

Where and when will the evaluation be conducted?

Note: The abstract should be on a separate page after the title page and before Section 1.

The proposed evaluation seeks to address the homelessness issue in Maryland. Point-in-time data showed that over 6,300 people in the state were homeless in 2020. While homelessness rates in the state have declined over the past decade, special populations such as veterans, chronically ill patients, and unaccompanied youth continue to be overrepresented among the homeless population. This evaluation seeks to evaluate the effectiveness of the Maryland Homelessness Solution Program (HSP) in reducing homelessness among Marylanders. The target population is individuals and households experiencing or at risk of homelessness in the State of Maryland. The evaluation will be conducted between January 2024 and December 2025 at 10 homeless shelters spread across Baltimore City, Baltimore, MD-BOS, Anne Arundel and Montgomery Counties. The following questions guide the evaluation: Compared to non-participants, are HSP participants more able to secure and maintain stable housing? How does the financial well-being of HSP participants compare with non-participants? Is the program accessible and relevant to the diverse needs of homeless people? Are program participants satisfied with the services offered? The evaluation will follow a case study with control group design. It will use survey and interview data collected from 300 randomly selected participants (150 HSP participants and 150 non-participants) at baseline, at program end, and at six months follow-up. Descriptive statistics will be used to analyze the program's effectiveness in reducing homelessness and to assess participants' satisfaction with available services. The evaluation findings will inform public health policy by providing insights on designing effective homelessness prevention interventions.

Section 1: Background/Rationale for Proposed Evaluation: This section should address the following elements (2-3 double spaced pages):

1. What public health problem and/or issue does the program/policy address?

2. Why is this problem or issue important?

3. What does existing data demonstrate about the magnitude and potential health impacts of this problem or issue?

4. What population(s) is affected by this public health problem/ issue?

5. What has previous evaluation literature found regarding this problem? Use peer-reviewed references to summarize the findings from evaluations of similar programs or policies. Address the following elements:

a. Describe your search methods, inclusion criteria, and studies that you reviewed

b. Describe the overall evaluation findings, including the strengths and weaknesses of this work

6. What are the gaps or limitations in the evaluation literature? How does your proposed evaluation address these issues?

The proposed evaluation focuses on homelessness. The 2009 Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act defines homelessness as a situation where a family or individual lacks an adequate, regular, and fixed nighttime residence (Substance Abuse and Mental Health Services Administration, SAMHSA, 2023). This definition of homelessness includes persons living in transitional housing, emergency shelters, abandoned buildings, the streets, and other places not meant for habitation. Individuals who stand to lose their nighttime residence within 14 days or are fleeing domestic violence and lack the resources to obtain alternative housing are also considered homeless under the HEARTH Act of 2009 (SAMHSA, 2023).

Data from the National Alliance to End Homelessness shows that the number of homeless people in the United States (US) has risen by approximately 6 percent since 2017 (SAMHSA, 2023). Contrary to expectations, the COVID-19 pandemic that peaked in 2020 had no significant effect on the US homelessness rate (National Alliance to End Homelessness, 2023). According to a report by the Maryland Interagency Council on Homelessness (2022), this is due to the increase in economic assistance programs and attempts to de-congregate shelters in line with social distancing requirements that took place at the time. Nonetheless, a point-in-time count conducted in 2022 showed that over 582,000 (9 in every 5,000) people were homeless across the United States (National Alliance to End Homelessness, 2023). Individual adults made up 72 percent of this population, while 28 percent lived in the streets and other unfit environments with their children (National Alliance to End Homelessness, 2023).

In Maryland, point-in-time count data from 2020 showed that slightly over 6,300 people were homeless statewide, with Baltimore City accounting for the highest rates of homelessness at 35 percent (Maryland Interagency Council on Homelessness, 2022). Generally, the statewide annual count showed that Maryland had approximately 28,288 homeless people in 2020 (Maryland Interagency Council on Homlessness, 2022). Unfortunately, homelessness disproportionately affects special populations, including persons with disability, mentally ill individuals, veterans, unaccompanied in-parenting youth, and victims of domestic violence (Maryland Interagency Council on Homelessness, 2022). Twenty-one percent of homeless people in the state, as per the 2020 point-in-time count, were chronically homeless individuals who had been homeless for over a year and struggled with a physical disability, substance use disorder, or severe mental illness (Maryland Interagency Council on Homelessness, 2022). Homeless veterans made up 7.5 percent of the homeless population, while individuals experiencing domestic violence and unaccompanied homeless youth made up 5 percent and 4 percent of the homeless population, respectively (Maryland Interagency Council on Homelessness, 2022).

Policymakers across the US need to institute ways of addressing homelessness because it increases the risk of mortality and reduces the average life expectancy of affected populations (Maryland Interagency Council on Homelessness, 2022). Studies have shown that homeless people report higher rates of injury-related death and violence than the general population (Maryland Interagency Council on Homelessness, 2022). Data from the Office of the Chief Medical Examiner (as cited in Maryland Interagency Council on Homelessness, 2022) shows that approximately 243 homeless people died in Maryland in 2020. This translates to a death rate of 3,835 per 100,000 people for homeless populations, which is three times the death rate of the general Maryland population (Maryland Interagency Council on Homelessness, 2022).

On average, homelessness reduces an individual's life expectancy by 30 years relative to the average person (Maryland Interagency Council on Homelessness, 2022). Unfortunately, homelessness is also associated with a higher risk of drug overdose and other complications related to the use of substances (Maryland Interagency Council on Homelessness, 2022). For instance, 65 percent of deaths reported among homeless individuals in 2020 were due to drug overdose and other related complications, with opioid overdoses accounting for 88 percent of these deaths (Maryland Interagency Council on Homelessness, 2022).

A significant amount of literature exists on the topic of homelessness. Many studies have sought to evaluate the effectiveness of various interventions in preventing or reducing homelessness. To identify possible studies for review, the evaluator conducted a literature search on three databases, Cochrane, PubMed, and Science Direct, for studies on homelessness prevention, case management, income assistance, and permanent supportive housing programs. The inclusion criteria included randomized controlled trials (RCTs) or systematic reviews conducted in high-income countries and published in peer-reviewed journals before 2020. Only studies that used housing stability as the outcome of interest were considered.

Generally, the studies found homelessness prevention programs and interventions effective in preventing and reducing homelessness. Dwyer et al. (2023) found that unconditional cash transfers, coupled with coaching supports and workshops on plan-making, goal-setting, and self-affirmation, improved housing stability for homeless people within three months better than cash transfers alone. Similarly, Philips and Sullivan (2022) found that financial assistance coupled with case management for homeless people increased housing stability and reduced evictions more than monetary assistance programs. The primary weakness of these evaluations is that they focus on a single intervention, income interventions, and disregard supportive housing programs, which are also quite common. In their systematic review, Aubry et al. (2020) address this weakness by studying the effect of both housing support and income assistance interventions. The study found that compared to usual care, housing support programs increase long-term housing stability at a rate of 1.13. At the same time, income assistance coupled with case management improved the number of days a homeless individual is housed by eight days relative to usual care (Aubry et al., 2020).

While these evaluations provide invaluable insights into the effectiveness of homeless assistance interventions, they have a limited scope, focusing on one or two interventions. The proposed assessment seeks to evaluate the HSP, which incorporates a range of diverse interventions targeted at homeless people. Moreover, the reviewed studies do not focus specifically on Maryland, and it would be interesting to check whether the findings of studies conducted in other environments could be replicated in the state of Maryland.

Section 2: Program or Policy Description: This section provides detailed information about the program/policy that you are evaluating. This section should include the following elements: (4-8 double spaced pages):

1. Program or Policy Purpose (Goals and Objectives)

a. What are the goals and objectives of the program/policy that you are evaluating? What are the goals and objectives of your evaluation? Goals are broad statements of what you want to achieve with the proposed evaluation. Objectives are specific steps that establish how the goal will be achieved. Objectives should be specific, measurable, achievable, relevant, and time-based (i.e., SMART). Both goals and objectives should be aligned with the main evaluation question(s).

Example of Goal and SMART Objective

Goal 1: Increase health educator knowledge about obesity among Hispanic

youth.

Objective 1.1: Conduct obesity training classes resulting in 80% of health educators completing the course Obesity in among Hispanic adolescents by May 31, 2021.

Note: Repeat this format if you have more goals/objectives.

The program to be evaluated is the Homelessness Solutions Program (HSP), which is administered by the Maryland Department of Housing and Community Development (DHCD) as a statewide response to the homelessness issue that continues to affect the state (HSP Policy Guide, 2022). The program provides funding and technical support to Local Homeless Coalitions (LHC) and Continuums of Care (COCs), coordinating resources and funding for local service providers serving the homeless population. To qualify for a grant under the HSP, an applicant must be a local government or non-profit entity offering a client-centered approach to homeless services and compliant with anti-discriminatory policies (HSP Policy uide, 2022). HSP grantees and sub-grantees must also implement formal grievance procedures, prove sound financial management of resources, and implement a sound homeless management information system (HMIS) (HSP Policy Guide, 2022).

The HSP pursues the following goals and objectives as stated in the HSP Policy Guide (2022) and the DHCD Consolidated Plan (2020), respectively:

Goal 1: Reduce the number of families/individuals at risk of homelessness by providing homeless prevention assistance and shelter diversion.

Objective 1.1: At least 50 percent of unsheltered homeless families/individuals reached through outreach programs exit to safe emergency shelters by the end of 2024.

Objective 1.2: 100 percent of households/individuals who fall homeless as a result of losing their permanent residence successfully exit to safe shelters by the end of 2024

Goal 2: Reduce the duration an individual or household remains homeless by supporting them in accessing permanent housing.

Objective 2.1: At least 80 percent of previously homeless individuals/households held in emergency shelters and interim housing options will transit effectively to permanent housing by the end of 2024.

Objective 2.2: At least 90 percent of households/individuals who fall homeless due to losing their permanent residence will successfully exit to alternative permanent housing by the end of 2024.

Goal 3: Reduce the number of individuals or households falling back to homelessness by connecting them to community-based resources, natural support networks, and income supports.

Objective 3.1: At least 10 percent of adult HSP beneficiaries report increased income by the end of 2024 due to the services and support they receive.

Objective 3.2: At least 25 percent of HSP beneficiaries report improved mental health by the end of 2024.

2. Need

a. What is the need for your evaluation? How do you expect it to contribute to public health?

While there has been a notable drop in homelessness rates in Maryland over the past several years, homelessness continues to disproportionately affect special populations, including veterans, people with physical disabilities, mentally ill individuals, and unaccompanied youth (Maryland Interagency Council on Homelessness, 2020). For instance, the 2020 point-in-time count revealed that veterans made up close to 8 percent of the homeless population, while chronically homeless persons with some disabling condition made up 21 percent of the homeless point-in-time count (Maryland Interagency Council on Homelessness, 2020). This data accentuates the need to evaluate the effectiveness of the HSP in reducing or preventing homelessness among special populations in Maryland. According to Smith and Ory (2014), program evaluation serves two purposes in public health promotion and education: to assess the effectiveness of a program or policy of interest and its interventions and to identify areas of weakness that could be used to improve overall program quality.

The proposed evaluation would help policymakers determine how effective the HSP is to the target population, including the relevance of its interventions to selected special populations. This would increase efficiency by ensuring program implementers identify interventions that work and commit more resources to make homelessness non-recurring, brief, and rare for all Marylanders (HSP Policy Guide, 2022). At the same time, the evaluation would help improve program quality by indicating the quality of deployed human resources, their areas of strength, and areas requiring capacity-building to enhance overall quality (Smith & Ory, 2014). Generally, the evaluation contributes positively to the field of public health by providing insights on what interventions may or may not work effectively in a homeless prevention program. The findings of the evaluation exercise will go a long way towards guiding program managers running similar programs on best practices, thus minimizing the risk of errors and wastage.

3. Program/Policy Context

a. What social, cultural, and political context/environment exists for the public health problem/issue that you propose to evaluate? What contextual/environmental factors may influence or affect your evaluation?

As Mabhala et al. (2017) point out, "Homelessness is a more complex social and public health phenomenon than the absence of a place to live" (p. 2). Homelessness is the result of a combination of structural forces (lack of access to mental health services, racial disparities, poverty, and lack of affordable housing) and social factors such as poor mental health, family breakdown, domestic violence, and addiction (Mago et al., 2013). Studies have shown that a majority of homeless people have been victims of traumatic experiences and social disadvantage from childhood, including dysfunctional families, neglect, emotional and sexual abuse, physical abuse, lack of psychological support, and disrupted schooling (Mabhala et al., 2017). All these factors serve as possible drivers of homelessness among Marylanders and may not be addressed by providing housing alone. The HSP's effectiveness will depend on how relevant its interventions are to the specific drivers of homelessness in the state. While the HSP may be effective in linking homeless people with interim or permanent homes, failure to effectively address the social context of homelessness may result in high rates of people falling back into homelessness upon exiting the program.

At the same time, structural factors such as ineffective housing assistance programs, budget cuts affecting federal housing, and rent increases may all increase homelessness rates (Mabhala et al., 2017). These factors are often dependent on the political environment and may affect the operations of the HSP, reducing its overall effectiveness. Thus, it would be prudent to assess the possible effect of these factors during the evaluation to obtain a holistic view of the program's sufficiency and relevance.

4. Target population

a. What population does the program/policy target?

Generally, the HSP targets the homeless population in Maryland. However, the program offers diverse support and services targeting different segments of the homeless population. The HSP's support for homeless individuals/households is divided into three areas: Outreach, Emergency Shelter, and Housing Stabilization (HSP Quick Reference Guide, 2022). Outreach services target unsheltered homeless people to link them with critical health services, housing, and emergency shelters (HSP Quick Reference Guide, 2022). Emergency Shelter services target unsheltered individuals living in places unfit for habitation, linking them with overnight shelters (HSP Quick Reference Guide, 2022). The HSP supports these populations indirectly by providing essential services during their stay in shelters and contributing towards the shelters' operating costs. Housing Stabilization focuses on preventing people from falling back into homelessness by supporting them in moving into permanent housing (HSP Quick Reference Guide, 2022).

The Housing Stabilization Services segment is divided into three support areas that target different populations (HSP Quick Reference Guide, 2022). The first area, Rapid Rehousing, targets households and individuals who lack adequate, regular, and fixed nighttime residences (HSP Quick Reference Guide, 2022). The second area, Homelessness Prevention, targets individuals with annual incomes below 30 percent of the median income. They risk becoming homeless but lack the support and resources to prevent it (HSP Quick Reference Guide, 2022). Finally, the Permanent Housing Case Management segment targets individuals in permanent housing who risk falling homeless for diverse reasons (HSP Quick Reference Guide, 2022).

5. Stage of Program or Policy

a. How long has the program/policy been in place?

b. Is it in the planning or implementation stage?

The HPS was instituted in 2017 following a decision by Marylands agencies to develop a central agency to coordinate activities of Local Homeless Coalitions (LHC) and continuums of Care (COCs) across the state (Maryland Interagency Council on Homelessness, 2022). The role of the HSP was to oversee homelessness programs statewide and to ensure that they align with federal and national best practices (Maryland Interagency Council on Homelessness, 2022). The formative years 2017 to 2019 served as the planning phase for the HSP. This period was characterized by setting goals and objectives, defining the project scope, and developing the DHCD Consolidated Strategic Plan 2020 to 2024. The program is currently in the implementation phase, guided by the framework, milestones, and targets laid out in the consolidated strategic plan 2020-2024 (DHCD Consolidated Strategic Plan, 2020).

6. Resources/Inputs

a. What resources are available to support your evaluation (e.g., staff, space, technology, money)?

The evaluation requires financial resources of at least $669,300.80. These cover staffing costs and benefits, staff training costs, and financing of equipment and supplies. In equipment and supplies, $43,000 will be required to cover emergency clothing or food vouchers, rental assistance vouchers, housing supplies and furniture vouchers, and participant incentives. Finances are necessary for staff salaries and benefits, amounting to $620,760.80, and an additional $5,600 is required to finance training sessions for case managers engaged in the evaluation.

The evaluation will engage two full-time staff and eleven part-time staff in human resources. The evaluation will employ a project director and project assistant on a full-time basis. The project director will serve as the primary contact person and will be responsible for recruiting case managers, training consultants, and the program analyst and overseeing the evaluation process. The project assistant will assist in accounting and record-keeping, data entry, and coordinating and scheduling training sessions. The part-time staff will include ten case managers and a program analyst. The case managers will be responsible for conducting assessments and offering case management services, while the program analyst will collect and analyze data.

The evaluation will additionally require office space from which the staff will carry out their duties. The DHCD agreed to provide office space (2,200 square feet) for the evaluation.

Finally, the evaluation team will require technological equipment, including computers (three desktop computers and one laptop for the project director), ten tablets for the case managers, an office printer, and a projector for meetings.

7. Outputs

a. What products/outputs will you produce as a result of the planned evaluation?

i) Outreach programs successfully identify and link homeless individuals/households with HSP services.

ii) The available assessments accurately identify homeless individuals income, housing needs, and financial literacy.

iii) Short-term financial assistance provided to homeless individuals and households successfully increases access to safe and affordable housing, which reduces their risk of homelessness.

iv) Long-term supportive and housing services provided to homeless individuals and households successfully increase access to permanent housing, which reduces the risk of falling back to homelessness.

v) Counseling and financial education services offered to homeless individuals effectively improves their mental health and financial literacy, which improves their ability to secure and maintain stable housing.

8. Outcomes

a. What are the evaluations intended outcomes (short-term, immediate, and long-term)?

b. How do these intended outcomes align with the overall programs or policys goal(s)?

a. Short Term Outcomes

i. Increased number of households/individuals receiving HSP services. An increase in the number of HSP beneficiaries would imply that the homeless population in the state is shrinking every year, and thus, fewer Marylanders are at risk of homelessness in each subsequent year.

ii. Program participants report increased housing stability. Increased housing stability would imply that beneficiaries have access to permanent homes, and run lower risks of falling back to homelessness.

iii. Program participants report improved financial literacy. Improved financial literacy would imply that HSP participants are able to make proper budgeting, investment, and financial management decisions, and are thus, more able to secure and maintain stable housing.

iv. Program participans report increased access to safe and affordable housing. Access to safe and affordable housing would imply that participants have links to homeless prevention assistance and shelter diversion programs, and hence run lower risks of being homeless on any given night.

v. Program participants report increased psychological well-being. Improved psychological wellbeing would imply that participants have improved mental health, can relate better with others, and can secure and maintain employment, thus minimizing their risk of homelessness.

b. Intermediate Outcomes

i. Program participants report reduced homelessness rates. Owing to increased access to shelter diversion, homelessness prevention, financial literacy, permanent housing, and counseling support programs, HSP participants would be able to secure and maintain stable housing, thus reducing their risk of being homeless. This would ultimately reduce overall homelessness rates in the state.

ii. Community members and stakeholders report increased understanding and knowledge of effective homelessness prevention interventions. Increased understanding of effective interventions would ensure that resources are channeled to strategies that effectively work to reduce homelessness, which would ultimately minimize the duration that individuals stay homeless and minimize the risk of beneficiaries falling back to homelessness.

iii. Local service and housing providers report increased collaboration. Increased collaboration among service providers working with homeless individuals would ensure that more homeless people, especially in marginalized areas, would benefit from HSP interventions, thus reducing overall homelessness rates in the state.

c. Long-term Outcomes

i. HSP participants report sustainable housing stability. Sustainable housing stability will imply that previously-homeless people can easily access homeless prevention programs and shelter diversion, thus reducing their risk of falling homeless at any given time.

ii. Inequality and poverty rates among HSP participants decline. Inequality and poverty have both been shown to be significant predictors of an individuals risk of homelessness. A decline in inequality and poverty rates will subsequently reduce the homeless population as at-risk individuals would report higher financial well-being and ability to secure permanent housing.

iii. HSP participants report improved overall quality of life. Improved quality of life would lead to better health and more happiness, thus reducing the risk of beneficiaries falling back to homeless due to poor psychological and physical wellbeing.

The HSP pursues three goals:

a) To reduce the number of families/individuals at risk of homelessness through the provision of homeless prevention assistance and shelter diversion.

b) To minimize the duration an individual or household remains homeless, support them in accessing permanent housing.

c) To reduce the number of individuals or households returning to homelessness by connecting them to community-based resources, natural support networks, and income supports.

9. Logic Model -- This is a graphical depiction of the components that you are evaluating and how they align with anticipated outcomes. [There is an example at end of document and template provided in the course] A logic model includes the following elements:

a. Inputs

b. Activities

c. Expected Outputs

d. Outcomes (short-term, intermediate, and long-term)

Inputs

Activities

Outputs

Outcomes

Short-Term

Intermediate

Long-Term

Funding allocated to HSP

Staff and volunteers trained to work with homeless participants

Partnerships with stakeholders

Access to affordable housing

Access to financial education resources

Evaluation specialist to assist with program planning and data collection

Outreach and engagement to identify and connect with individuals and families experiencing homelessness

Assessment of participants' needs, including housing, income, and financial literacy

Delivery of short-term financial assistance, such as rental and utility payments, to prevent or end homelessness

Delivery of long-term housing and supportive services to help participants maintain housing stability and improve their financial situation

Delivery of financial education and counseling services to help participants build financial literacy and achieve their financial goals

Outreach programs successfully identify and link individuals and households experiencing homelessness to HSP services

The available assessments accurately identify homeless individuals income, housing needs, and financial literacy

Short-term financial assistance provided to homeless individuals and households successfully increases access to safe and affordable housing.

Long-term housing and supportive services delivered to homeless individuals and families successfully increase access to permanent housing.

Counseling and financial services offered to homeless individuals effectively improve their mental health and financial literacy.

Increased number of individuals and families receiving services through HSP

Increased housing stability for program participants

Improved financial literacy for program participants

Increased access to affordable and safe housing for program participants

Improved psychological well-being for program participants.

Reduced rates of homelessness for program participants in Maryland

Increased knowledge and understanding of effective homelessness solutions among stakeholders and community members

Increased collaboration among housing and service providers to address homelessness in Maryland

Sustainable housing stability and financial well-being for HSP participants

Reduced rates of poverty and inequality among HSP participants.

Improved overall quality of life among HSP participants

Section 3: Evaluation Focus: This section provides information on how you will design your evaluation. This section should include the following elements (1-2 double spaced pages):

1. Evaluation Question(s):

a. What specific questions do you intend to answer through your evaluation?

The proposed evaluation seeks to answer the following crucial questions:

a) Compared to non-participants, are HSP participants more able to secure and maintain stable housing?

b) How does the financial well-being of HSP participants compare with non-participants at program start, end, and at six-month follow-up?

c) Is the program accessible and relevant to the diverse needs of homeless people?

d) Are program participants consistently satisfied with the support and services they receive from the HSP?

2. Stakeholders:

a. Who are the stakeholders? Who will use the evaluation findings?

b. What role will stakeholders play in developing this evaluation proposal?

c. How do you plan to engage these stakeholders when implementing this evaluation proposal (e.g., participating in data collection, interpetation of findings)?

The stakeholders in this evaluation can be categorized into primary and secondary stakeholders. Primary stakeholders have a direct stake in the evaluation, and include HSP participants, the DHCD board, and staff working directly in the HSP. Secondary stakeholders are indirectly affected by the evaluation findings. They include HSP grantees and sub-grantees (the Local Homeless Coalitions (LHC) and Continuums of Care (COCs) that work with the HSP), funders, and local communities.

The evaluation findings will be crucial to the DHCD and its staff, grantees and sub-grantees, funders of the HSP, and local communities. To the DHCD and its staff, the evaluation findings will provide insights into the overall quality and effectiveness of the HSP's homelessness solutions and possible areas of improvement. This will help them determine what works and what does not work effectively and identify areas of improvement to facilitate the realization of policy goals. The HSP management could use the evaluation findings to analyze program staff performance and identify areas that require capacity enhancement (Harris, 2016). Grantees and sub-grantees of the HSP interact directly with the HSP participants. The evaluation findings will indicate how relevant the services and supports offered by grantees and sub-grantees are. It will also indicate how satisfied their beneficiaries are with the services and support they receive. To the federal government, the primary funder of the HSP, the evaluation findings will indicate whether or not the program is worthwhile by measuring its effectiveness in reducing homelessness in Maryland.

Harris (2016) notes that engaging stakeholders in program evaluation goes a long way toward creating buy-in and minimizing resistance in case there is a need for changes to enhance program quality and effectiveness. Thus, stakeholders will be engaged in developing the evaluation proposal and its implementation. In developing the proposal, stakeholders, particularly program staff, volunteers, grantees, and sub-grantees, will design data collection tools and determine what to include in the survey/interviews to assess the quality of HSP interventions. During the implementation phase, stakeholders such as representatives of COCs, LHCs, and program staff will be engaged in interpreting and drawing conclusions from the evaluation findings. Finally, a stakeholder forum will be organized with representatives of all relevant stakeholders to disseminate the project findings. The stakeholders will be expected to disseminate the evaluation findings to their colleagues in their respective agencies and organizations.

3. Evaluation Design:

a. What is the design for this evaluation (e.g., experimental, pre-post with comparison group, time series, case-study, post-test only)?

b. Why was this design selected? What are the strengths and limitations of this design?

The evaluation uses a case study with a control group design. According to Schoch (2020), a case could be a nation, community, organization, small group, or individual (Schoch, 2020). HSP participants serve as the cases for this analysis. The case study design is appropriate for this evaluation as it provides a platform to study the HSPs interventions and assess their effectiveness within the programs real-life context (Schoch, 2020). Schoch (2020) advocates for the case study design when a researchers aim is not to generalize findings, but to obtain an in-depth look at a program, organization, or individual and to get a deep understanding of its inner workings. The proposed evaluation not only focuses on measuring the effectiveness of the HSP, but also gaining an in-depth understanding of its interventions and their relevance to the needs of homeless individuals through surveys, interviews, document reviews, and observations. As such, the case study design is highly appropriate for the proposed evaluation. The use of the control group will be crucial in assessing whether an association actually exists between an exposure (HSP participation) and an outcome (reduced risk of homelessness). The inclusion of the control group provides a means to compare outcomes between cases and controls.

One of the greatest strengths of the case study design is that it helps a researcher focus on a specific case within the confines of time and space, which provides a deep understanding of the case and its inner workings (Schoch, 2020). Moreover, the design allows for the collection of different types of data using a combination of methods, from surveys, to observations, documents, and interviews to obtain an in-depth view of the case (Schoch, 2020). The greatest weakness of the case study design is that its findings are not generalizable to other cases (Schoch, 2020). For instance, the findings of this evaluation may not be generalizable to other homelessness prevention programs in other states, since the evaluation confines itself to the HSP and its inner workings. Another major weakness of the selected design is the high risk of selection bias when selecting the controls (Tenny et al., 2023). According to Tenny et al. (2023), controls must be similar to the cases in many aspects, with an equal risk of developing the outcome as the cases. Selection of controls that are not similar to the cases would result in selection bias and misleading findings.

The evaluation will compare the outcomes of HSP participants (cases) with non-participants (controls) to determine whether the program effectively reduces the risk of homelessness among its beneficiaries. The control group will include homeless individuals who frequent the participating shelters but are not registered to receive HSP services. Comparison will be based on financial literacy and housing stability scores before joining the program, when one completes the program, and six months after exiting the program. This will enable the evaluators to assess the progression in the risk of homelessness due to participating in the HSP.

Section 4: Data Collection: This section provides information on how you will collect/compile data for your evaluation. It should provide information on the methods that you will use to collect data and how these methods, and the data that will be compiled, are related to the evaluation question(s) you are trying to answer. This section should include the following elements: (1-2 double spaced pages):

Data Collection Methods:

1. Will you collect new data to answer the evaluation questions, or will you use secondary data?

2. How will the data align with relevant program/policy performance measures?

3. What methods will be used to collect or acquire the data?

4. Will you collect data from a sample of participants? If so, how will the sample be selected?

5. How will data collection instruments be identified and tested? If you are using a previously validated instrument, include this detail and cite the corresponding source(s).

6. How will the quality and utility of data be determined?

7. From whom or from what will data be collected? What is the source of the data?

8. How will the data be stored, managed, and protected?

9. Address ethical considerations (e.g., participant informed consent, confidentiality, IRB approval)Primary data will be collected through a combination of qualitative and quantitative techniques. A survey will be administered to sampled homeless adults at the program's start, at the time of exit, and six months after exit to gather quantitative data on satisfaction with HSP services, financial literacy, and housing stability. Financial literacy and housing stability will measure one's risk of homelessness thrice. Face-to-face interviews will be conducted to collect qualitative data from 10 HSP participants and 10 non-participants on the program's accessibility and relevance to the needs of homeless individuals. Studies have suggested that 20 to 30 interviews are optimal for theoretical saturation when conducting semi-structured interviews (Vasileiou et al., 2018). The evaluators feel that 20 interviews (10 from cases and 10 from controls) will sufficiently provide the information needed with minimal strain on the available resources. The interviews will be conducted at the end of the program (the time of exit). Thus, interviews will be used to collect in-depth information on the attitudes and perceptions of both HSP participants and non-participants. At the same time, the survey will provide data on the HSP's effectiveness in reducing homelessness among its participants. The evaluators will also review program records to gather data on the number of homeless individuals served annually and the support available for beneficiaries.

The sample will consist of 300 homeless adults, 150 of whom will be HSP participants, and 150 will be non-participants. The sample will be selected randomly from program records of community-based organizations and homeless shelters serving homeless individuals. Random sampling will help enhance the objectivity of the study findings (Bloomfield & Fisher, 2019). To be eligible to participate, individuals must be frequent visitors at the participating shelters, be experiencing homelessness, and be within the set demographics.

To enhance the reliability and validity of the evaluation findings, the evaluators will use previously validated instruments to measure participants' satisfaction with HSP services, financial literacy, and housing stability. Satisfaction with HSP services will be measured using a survey adapted from the Client Satisfaction Questionnaire (CSQ-8) (appendix C in the appendices). Studies testing the psychometric properties of the CSQ-8 in outpatient and residential settings have found the instrument to be a reliable and valid measure of client satisfaction (Pedersen et al., 2022). Pedersen et al. (2022) found the CSQ-8 to have high internal consistency (Cronbach Alpha = 0.95) and high factor validity, with all factor loadings greater than 0.8. On their part, Kelly et al. (2017) found the CSQ-8 to have high concurrent validity, having shown a strong correlation with other scales, specifically the Treatment Perceptions Questionnaire (TPQ).

Financial literacy will be measured using a survey adapted from the Financial Well-being Scale (appendix D in the appendices) (Consumer Financial Protection Bureau, 2023). Quantitative studies by the Consumer Financial Protection Bureau (2023) found that the financial well-being scale had high internal consistency, as shown by the Cronbach Alpha of 0.8. This makes it a valid and reliable measure of financial well-being. Finally, housing stability will be measured using a survey adapted from the Vulnerability Index-Service Prioritization Decision Assistance Tool (VI-SPDAT v 2.0 for individuals) (included in the appendices). Studies have found the tools predictive validity and test-retest reliability to be relatively lower than other vulnerability assessment tools (Brown et al., 2018). However, the study found that the tools predictive ability was improved by addition of housing type questions (Brown et al., 2018). The VI-SPDAT V 2.0 is preferred for the proposed evaluation because it focuses hugely on housing type and thus gives a better view of housing stability as compared to other vulnerability tools that focus more on social relationships and mental health.

To ensure the data collected is of high quality, the evaluation will standardize the data collection process by ensuring that all surveys are conducted and filled out in English. Interpreters will be engaged for non-English-speaking participants to ensure they adequately understand the interview/survey questions and that the data they give is captured accurately. Case managers engaged in data collection will be trained before the evaluation to ensure that they are well-versed with the survey instrument/interview protocol and adequately understand their roles and the rights of participants.

Participants will be required to give their consent before the start of the evaluation. They will sign an informed consent form to indicate their consent to participate. Participants will be notified in the consent form that their participation is voluntary and that any information they provide will be used solely for evaluation purposes. In line with anonymity requirements, self-generated identity codes that cannot be traced to a specific individual will be used to identify participants across the three waves of data collection (Audette et al., 2020). The self-generated codes will be created from answers to four personal questions arranged in a pre-determined order: the first letter of the participants middle name, their month of birth, how many older sisters they have, and the first initial of their mothers first name (Audette et al., 2020). Thus, a participant whose middle name is Cate (C), was born in April (04), has two older sisters (02), and whose mother is named Mary (M) would generate the code C0402M, which will be used to link their data across the three waves of data collection. To ensure confidentiality, all the collected data will be stored electronically in a password-protected laptop used solely for the evaluation under the program director's care. The program manager will obtain prospective IRB approval to evaluate and enhance the welfare and rights of participants as human subjects.

10. Evaluation Question Link:

a. How does each data collection method relate to the evaluation question(s) proposed? Suggested table:

Evaluation Question

Data Collection Method

Source of Data

Timeline for Data Collection

1.Compared to non-participants, are program participants more able to secure and maintain stable housing?

Survey

HSP participants (150) and non-participants (150)

February 2024 March 2024 (baseline);

November 2024 to February 2025 (end of program); August to September 2025 (follow up)

2.How does the financial well-being of HSP participants compare with non-participants at program start, end, and at six-months follow-up?

Survey

HSP participants and non-participants

February 2024 March 2024 (baseline);

November 2024 to February 2025 (end of program); August to September 2025 (follow up)

3.Is the program accessible and relevant to the diverse needs of homeless people?

Face-to-Face Interviews

10 HSP Participants and 10 non-participants

February 2025 to March 225

4. Are program participants consistently satisfied with the support and services they receive from the HSP?

Survey

Face-to-Face Interviews

HSP participants (150)

HSP participants (10)

November 2024 to February 2025 (end of program)

February 2025 to March 2025

Section 5: Data Analysis and Interpretation: In this section, you will provide information on the standards you will use to judge the performance and/or outcomes of the program or policy that you evaluate. You should describe how you will analyze your evaluation findings and interpret and justify your conclusions. (1-2 double spaced pages):

Indicators and Standards:

1. What are the measurable or observable elements that can tell you about the performance or outcomes of the program or policy that you are evaluating?

2. What constitutes success (i.e., by what standards will you compare your evaluation findings?) Suggested table:

Evaluation Question

Criterion or Indicator

Standards (i.e., what constitutes success?)

1. Capacity of the HSP

HSP beneficiaries who secure housing units every year as a proportion of the total population of beneficiaries according to agency records.

Number of new partnerships with other housing support networks every year

At least 30 percent of HSP beneficiaries acquire permanent housing units every year

At least 25 new partnerships are formed with other housing support organizations to increase the HSPs geographical reach.

2. Reduce the rate of homelessness

Annual increases in the number of homeless individuals and families joining the HSP

The population of homeless individuals served by the HSP increases by 10 percent annually.

3. Reduce time spent unsheltered

Proportion of homeless individuals living in conditions unfit for habitation that are referred to HSP-affiliated emergency shelters and diversion programs

75 percent of homeless individuals living in areas unfit for human habitation are referred to emergency shelters and diversion programs.

Analysis of Data:

1. What methods will you use to analyze your data (e.g., descriptive statistics, inferential statistics)?

2. Provide example data table templates, if applicable.

The collected data will be analyzed using qualitative and quantitative techniques. Survey data will be analyzed using descriptive statistics to assess financial literacy and housing stability changes over time. Percentages and means will determine the proportion of HSP participants with a satisfaction score above 26 in the Client Satisfaction Questionnaire. The Client Satisfaction Questionnaire yields scores of between 8 and 21, with scores above 26 indicating high levels of satisfaction. Percentages will also be used to determine the proportion of participants with above average scores in financial literacy and housing stability across the three waves of data collection. These will then be compared with those of non-participants.

One-way analysis of variance (ANOVA) will be used to examine whether participating / not participating in HSP (independent variable) affects financial literacy and housing stability (dependent variables) by comparing participants mean scores against non-participants at baseline, program end and six-months follow up. One-way ANOVA is appropriate for examining mean differences between two or more populations when the dependent variable is continuous and the independent variable is categorical as in this case (Astle et al., 2023). Table 1 below presents a template of the one-way analysis of variance table comparing participants' and non-participants' mean scores in financial literacy and housing stability. Paired samples t-tests will be conducted to determine whether HSP participants' financial literacy and housing stability scores at program end and at six-month follow up differ significantly from baseline levels (Astle et al., 2023). Table 2 below presents a sample of the paired t-tests sample results table.

Thematic analysis will be used to analyze data collected through interviews. Thematic analysis will identify recurrent themes related to cultural competence and program effectiveness. The evaluator will also review program records to identify trends and patterns in the HSP support and service offerings. The results of all three analysis techniques will be integrated to provide a comprehensive view of the HSP's impact.

Table 1: One-way Analysis of Variance Results Table Template

Sum of Squares

df

Mean Square

F

Sig.

Between Groups

Within Groups

Total

Table 2:...