Embarking on College and Career: Interim Evaluation of Urban Alliance

R E S E A R C H R E P O R T Embarking on College and Career Interim Evaluation of Urban Alliance Brett Theodos Michael R. Pergamit Devlin Hanson Sara Edelstein Rebecca Daniels May 2016 M E T R O P O L I T A N H O U S I N G A N D C O M M U N I T I E S P O L IC Y C E N T E R A B O U T T H E U R B A N I N S T I T U T E The nonprofit Urban Institute is dedicated to elevating the debate on social and economic policy. For nearly five decades, Urban scholars have conducted research and offered evidence-based solutions that improve lives and strengthen communities across a rapidly urbanizing world. Their objective research helps expand opportunities for all, reduce hardship among the most vulnerable, and strengthen the effectiveness of the public sector. A B O U T T H E S O C I A L I N NO V A T I O N F U N D This report is based upon work supported by the Social Innovation Fund (SIF), a key White House initiative and program of the Corporation for National and Community Service (CNCS). The Social Innovation Fund combines public and private resources to grow the impact of innovative, community-based solutions that have compelling evidence of improving the lives of people in low-income communities throughout the United States. These findings are preliminary and the full report is under review by the Corporation for National and Community Service.Copyright © May 2016. Urban Institute. Permission is granted for reproduction of this file, with attribution to the Urban Institute. Cover image from Urban Alliance. Contents Acknowledgments vi Executive Summary vii What is the Urban Alliance program model? vii What was the study design? viii Who were the youth in the study? x What services did youth receive? xi What impacts did the program generate? xiii Introduction 1 Background 3 Barriers to Education 3 Programs to Promote Success 4 Evidence from Youth Programs 5 Work Experience 5 Work-Based Learning 6 Job Training 8 Case Management and Mentoring 8 College Access and Readiness 9 Urban Alliance High School Internship Program Model 11 Target Population and Recruitment 11 Program Components 12 Training: Pre-work and Workshop 12 Paid Internships 13 Coaching and Mentorship 13 Alumni Services 14 Logic Model 14 Evaluation Design 17 Random Assignment 17 Data Collection 17 Application Baseline Data 18 Aggregated High School Baseline Data 18 Neighborhood Baseline Data 18 Youth-Level High School Baseline and Outcome Data 19 Program Implementation Data 19 Interim Outcome Survey Data 19 Postsecondary Institution Outcome Data 20 Analysis Methods 20 Predicting Treatment Take-Up 20 Intent to Treat Analysis 21 Treatment on the Treated 22 Study Participants 24 Overview of Baseline Characteristics 24 Differences between Treatment and Control Groups 29 Differences across Sites 29 Program Take-Up and Services Received 32 Program Attrition 32 Take-Up Regressions 33 Reasons for Attrition 36 Services Received 37 Program Impacts 40 Education and Employment Preparation 40 Education Achievement 43 High School 43 College Attendance and Quality 44 Employment, Wages, and Savings 47 Implications for Practice and Policy 49 Target Population 49 Program Attrition 50 Self-Selection into the Program 51 Schools Matter 52 Gender Differences 52 Unexplained Differences 53 Unknown Differences 53 Is the Urban Alliance an Employment Program, a College-Access Program, or Both? 54 Conclusion 54 Appendix A: Survey Methodology 56 Appendix B: Differential Attrition 59 Appendix C: Full Sample and Subgroup Impact Tables 61 Appendix D: Survey Instrument 85 Notes 118 References 120 About the Authors 123 Statement of Independence 125 VI A C K N O W L E D G M E N T S Acknowledgments This report was funded by Corporation for National and Community Service’s Social Innovation Fund. We are grateful to them and to all our funders, who make it possible for Urban to advance its mission. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at www.urban.org/support. The authors thank the many people who made this report possible. We are grateful for the support of the Corporation for National Service’s Social Innovation Fund, Venture Philanthropy Partners’ YouthConnect initiative, the World Bank Group’s Community Outreach Program, and the Kellogg Foundation. Special thanks to the dedicated directors and staff at Urban Alliance, in particular Eshauna Smith, Daniel Tsin, Jose Sousa, and Meaghan Woodbury. Thanks also to Sean Segal and Veronica Nolan, formerly of Urban Alliance, for their help designing and initiating this research study. In addition, we thank Helen Ho, Taz George, Lesley Freiman, and Elizabeth Davies for their assistance with several components of this research study. E X E C U T I