Evaluation Guidebook for Projects Funded by STOP Formula Grants Under the Violence Against Women Act

EVALUATION GUIDEBOOKFOR PROJECTS FUNDEDBY S.T.O.P. FORMULA GRANTSUNDER THE VIOLENCE AGAINST WOMEN ACTBy:Martha R. BurtAdele V. HarrellLisa C. NewmarkLaudan Y. AronLisa K. Jacobsand othersDecember 1997Urban Institute2100 M Street, N.W.Washington, DC 20037(202) 833-7200http://www.urban.orgThis project was supported by Grant No. 95-WT-NX-0005 awarded by the National Institute ofJustice, Office of Justice Programs, U.S. Department of Justice. Points of view in this documentare those of the authors and do not necessarily represent the official position or policies of theU.S. Department of Justice or of other staff members, officers, trustees, advisory groups, orfunders of the Urban Institute. iTABLE OF CONTENTSTABLE OF CONTENTS.....................................................iWHERE TO GET RESOURCES DISCUSSED IN THIS GUIDEBOOK ................viiPREFACE................................................................ixCHAPTER 1: GETTING STARTED: THINKING ABOUT EVALUATION............. 1What Should Be Evaluated?..............................................2Working Together: Participatory Evaluation.................................4Respect for Women’s Privacy and Safety....................................5How to Use this Guidebook..............................................5CHAPTER 2: DEVELOPING AND USING A LOGIC MODEL...................... 7Creating a Logic Model.................................................8EXHIBIT 2.1: Logic Model for Evaluation of Shelter-Based Services For Domestic Violence Victims.........................................9EXHIBIT 2.2: Logic Model for Your Program..............................10Program Evaluability..................................................12If Not Impact Evaluation, What?.........................................14Evaluation Options for Programs that Are Already Under Way..................14CHAPTER 3: MAKING EVALUATION WORK FOR YOU.........................15Finding an Evaluator..................................................15Getting What You Want from Evaluation...................................17For State Coordinators.................................................18For Subgrantees......................................................20CHAPTER 4: USING EVALUATION INFORMATION............................23Uses for Program Promotion............................................23Avoiding Misunderstandings of Your Program.........................23Using Evaluation Information to Get Funding..........................25Using Evaluation Information to Increase Public Appreciation of Your Program..................................26Uses for Program Development..........................................26Uses for Community Development........................................29Collecting Data over Time........................................30CHAPTER 5: USING THE SAR AND THE SSS..................................31What Are the SAR and the SSS?.........................................31Getting Information for the SAR and the SSS...............................31 iiUsing the SAR and SSS Information......................................33State-Level Monitoring and Planning................................33Monitoring and Modifying the Subgrant..............................34Providing Feedback on the Forms...................................34How to Handle Special Circumstances.....................................35Projects with Several Funding Sources...............................35Projects that Enhance Services to the Same Victims.....................35Projects that Enhance Service Quality but NOT Numbers.................36Other Ways to Get Useful Information.....................................37The SAR Form......................................................38The SSS Form.......................................................41CHAPTER 6: CHOOSING AN EVALUATION DESIGN...........................51Impact Evaluation Designs..............................................52Non-Experimental Impact Evaluations...............................52Quasi-Experimental Designs.......................................55Experimental Designs............................................57The Impact Evaluation Design “Decision Tree”........................59EXHIBIT 6.1: Decision Tree......................................60Process Analysis.....................................................61Key Elements..................................................61Design Variations...............................................61Practical Issues.................................................62Performance Monitoring...............................................62Key Elements..................................................62Design Variations...............................................63Considerations/Limitations........................................63Practical Issues.................................................64Data Collection Strategies..............................................65Q