The Synthetic Control Method as a Tool to Understand State Policy

S T A T E A N D L O C A L F I N AN C E I N I T I A T I V E R E S E A R C H R E P O R T The Synthetic Control Method as a Tool to Understand State Policy Robert McClelland Sarah Gault March 2017 A B O U T T H E U R B A N I N S TI T U T E The nonprofit Urban Institute is dedicated to elevating the debate on social and economic policy. For nearly five decades, Urban scholars have conducted research and offered evidence-based solutions that improve lives and strengthen communities across a rapidly urbanizing world. Their objective research helps expand opportunities for all, reduce hardship among the most vulnerable, and strengthen the effectiveness of the public sector. Copyright © March 2017. Urban Institute. Permission is granted for reproduction of this file, with attribution to the Urban Institute. Cover image by Tim Meko. Contents Acknowledgments iv The Synthetic Control Method as a Tool to Understand State Policy 1 The Synthetic Control Method 2 ADH’s Analysis of California’s Tobacco Control Program 3 Procedures 5 Output 8 Sensitivity Tests 14 Outcome Lags 15 Nonlag Predictor Variables 19 The Year Range for Averaging Predictors 23 Selecting Predictor Weights 25 Abadie, Diamond, and Hainmueller’s In-Time Placebo and Leave-One-Out Tests 28 Conclusion 32 Appendix A. Economic Development Applications 34 Castillo and Coauthors’ Analysis of Salta’s Tourism Development Policy 34 Ando’s Analysis of Japanese Nuclear Power Facilities 35 Appendix B. Synth Package Predictor Weight Selection 37 Notes 38 References 39 About the Authors 40 Statement of Independence 41 IV A C K N O W L E D G M E N T S Acknowledgments This report was funded by the Laura and John Arnold Foundation. We are grateful to them and to all our funders, who make it possible for Urban to advance its mission. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at www.urban.org/support. We gratefully acknowledge comments and suggestions from Norton Francis, Kim Rueben, and Rob Santos, and editing from the Urban Institute communications team. The Synthetic Control Method as a Tool to Understand State Policy Identifying good governance depends on accurately evaluating policies for their efficiency and effectiveness, and many tools are available for that purpose. Case studies of regional economies are an often-used example in which policies are evaluated through detailed examinations of economic conditions before and after the policies are implemented. Because those analyses are qualitative rather than quantitative, rigorously identifying comparison groups whose outcomes can be contrasted with the outcome of the region undergoing the policy change is difficult. But without those control groups, separating the policy’s effect from the effects of nonpolicy variables is nearly impossible. Nearby regions are sometimes used as controls for lack of alternatives, but geographic proximity is a poor metric for similarity if regions have substantial differences in political or cultural environments. Further, policies may spill across borders, confounding any comparison. The qualitative approach also impedes the analyst’s attempt to generalize the analysis beyond the case at hand because few quantitative results can be applied to similar situations. Measures of similarity can be difficult to define, and no measures of statistical precision or accuracy exist. An increasingly popular method for policy evaluation, the synthetic control method (SCM), addresses those problems. It provides quantitative support for case studies by creating a synthetic control region that simulates what the outcome path of a region would be if it did not undergo a particular policy intervention. The SCM creates this hypothetical counterfactual region by taking the weighted average of preintervention outcomes from selected donor regions. The donor regions that combine to form the synthetic control are selected from a pool of potential candidates. Predictor variables that affect the outcome and the outcome variable itself before the policy is enacted determine the selection of donor regions and weights. The resulting synthetic closely matches the affected region’s outcome before policy enactment and is a control for the affected region following enactment. After policy enactment, the difference in outcomes between the affected region and its synthetic control counterpart reveals the policy’s effectiveness. The SCM is transparent. Analysts can evaluate how well the synthetic control’s outcome matches the affected region’s outcome before the policy change. In addition, donor regions and the weights assigned to them are known, and analysts can evaluate those regions’ similarity to the policy region. It