Adaptation of the CDC Evaluation Framework to Design the Evaluation of Reaching Teens, a Multi-Institutional, Region Wide Teen Resiliency Training Program

Purpose: Mental Health Connection of Tarrant County (MHC) is a collaboration of public and private agencies working together to revolutionize the mental health service delivery system in Tarrant County, Texas. MHC is partnering with the American Academy of Pediatrics (AAP) on a multi-institutional region‐wide staff training program. This program, Reaching Teens, uses the teen resiliency curricula developed by Dr. Kenneth Ginsburg, a pediatrician specializing in Adolescent Medicine. The goals of reaching Teens are to develop a trauma‐informed workforce, improve system integration and increase the use of evidence-informed practices in youth behavioral care in Tarrant County. Our research purpose was to adopt the Centers for Disease Control and Prevention (CDC) evaluation framework to planning the evaluation of Reaching Teens. The CDC evaluation framework engages stakeholders in the evaluation with the goal of stimulating innovation that leads to improving outcomes and detecting program effects. Methods: We engaged agency stakeholders (Step 1), described the program using a logic model (Step 2), and focused the evaluation design on the issues most important to MHC and the organizational partners (Step 3). We conducted a baseline assessment of knowledge, attitudes and beliefs about teen resiliency and approaches for successfully working with adolescents in the mental health system. Results: Our results consist of: (1) a logic model and (2) baseline survey findings. The Reaching Teens logic model is a roadmap to guide the ongoing planning, program activities and evaluation. With stakeholders, we prioritized three program components: (1) staff development; (2) organizational capacity building; and (3) system‐wide collaboration. We identified key program outcomes: (1) engaging youth; (2) boundary-setting; (3) trauma-informed care; (4) commitment to “care” factor; (5) strengths-focussed; (6) resiliency; (7) job satisfaction; (8) system recognized for effective communication with youth; and (9) decreased turnover. Key baseline survey findings (n=438 staff participants from 15 agencies) are: high levels of job satisfaction (4.2 - 4.5, 5-point scale), congruence of personal and organizational values (3.3 - 4.0, 5-point scale), average teamwork in workplace (3.4 - 4.0, 5-point scale). Conclusion: The Reaching Teens logic model is a powerful tool to guide system-wide collaboration toward the goal of more effectively reaching adolescents in mental health care. There are three program components: staff development, building organizational capacity, and strengthening system-wide collaboration. Baseline levels of the outcome measures (job satisfaction, teamwork, and value congruity) are relatively high among participating staff; thus, it may be difficult to demonstrate program impact by increasing these scores post-intervention.