Validity Evidence for ACT Compass Placement Tests

*050202140* Rev 1Validity Evidence for ACT Compass® Placement TestsPaul A. WestrickJeff AllenApril 2014ACT Research Report Series2014 (2) For additional copies, write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243-0168© 2014 by ACT, Inc. All rights reserved. Validity Evidence for ACT Compass® Placement Tests Paul A. Westrick Jeff Allen ii Abstract We examined the validity of using Compass® test scores and high school grade point average (GPA) for placing students in first-year college courses and for identifying students at risk of not succeeding. Consistent with other research, the combination of high school GPA and Compass scores performed better than either measure used alone. Results also indicate that, relative to Compass scores, the predictive strength of high school GPA decays with student age. We recommend using multiple measures for making course placement decisions and identifying students for intervention. iii Acknowledgements The authors thank Nancy Peterson, Edgar Sanchez, James Sconing, and Richard Sawyer for their helpful comments and suggestions on earlier drafts of this report. Validity Evidence for ACT Compass® Placement Tests Introduction ACT Compass assessments measure students’ skills in reading, writing, and mathematics. They are designed to assist institutions in placing college students in appropriate credit-bearing courses or in developmental or preparation courses. Compass scores can also be used to identify students who are enrolled in credit-bearing courses, but who may need additional academic support, such as tutoring or supplemental instruction. The Compass placement assessments are listed in Table 1, along with the number of assessments given at two- and four-year institutions in the 2011-2012 year (ACT, 2012). Table 1 Number of Compass Assessments Administered at Two- and Four-Year Institutions, 2011-2012 Compass test N Writing Skills 1,350,835Reading 1,456,301e-Write (writing essay) 185,439Numerical Skills/Pre-algebra 1,191,390Algebra 1,212,731College Algebra 142,892Trigonometry 72,585Geometry 25,099 In this study, we evaluate the validity of using Compass placement test scores for making placement decisions and identifying students for additional academic support. We present three types of evidence: 1. Statistical measures of the overall strength of the relationship between the predictors – Compass scores and high school grade point average (HSGPA) – and success in standard first-year courses. This evidence is aligned to predicting success in standard first-year college-level courses. 2 2. Accuracy rates that estimate the proportion of students who are accurately placed into standard courses (likely to succeed in the standard courses) or developmental courses (unlikely to succeed in the standard courses). This evidence is related to placing students in developmental or college-level courses. 3. Intervention hit rates that measure the accuracy of identifying the students least likely to succeed in standard courses. This evidence is related to identifying students in college-level courses who are at risk of not succeeding. We present this evidence for the five most frequently used Compass assessments – Writing Skills, Reading, Numerical Skills/Pre-Algebra, Algebra, and College Algebra. Previous studies have reported that Compass placement test scores and HSGPA, when used together, provide greater accuracy than either measure used alone (Noble, Schiel, & Sawyer, 2004; Scott-Clayton, 2012). This implies that institutions should use multiple measures (e.g., HSGPA and Compass scores), when possible. To accommodate the use of multiple measures for course placement, the Compass system offers a feature for weighting HSGPA and other measures, along with Compass scores, to create an overall placement score (ACT, 2006a). Among the institutions that use Compass, however, many students do not have high school grade information, either because they do not have a high school diploma1 or because an official high school transcript is not readily available. Thus, single-predictor models are also used in practice and warrant research. This study considers three placement models: 1. Compass subject area score only, 2. HSGPA only, and 1 Among Compass examinees tested between September 1, 2011 and August 31, 2012 who reported their type of high school certificate, 62% reported having a high school diploma, 20% reported still being in high school, 13% reported having a GED, 4