Relating ACT Aspire Scores to Performance in High School Courses

Relating ACT Aspire Scores to Performance in High School Courses Jeff Allen, PhD, Justine Radunzel, PhD, and Jizhi LingResearch Report2019-5 ABOUT THE AUTHORSSUMMARYSO WHAT?NOW WHAT?Jeff Allen, PhDWe examined the relationships of ACT Aspire test scores and high school course grades, AP test scores, and Partnership for Assessment of Readiness for College and Careers (PARCC) test scores, demonstrating criterion-related as well as convergent and discriminant validity evidence. We found that ACT Aspire test scores are strong predictors of success in standard, career-focused, AP, and dual enrollment courses. Moreover, ACT Aspire scores demonstrate expected convergent and divergent relations across grade levels and with other standardized measures of college and career readiness.The results are similar across student subgroups and support five intended uses of ACT Aspire test scores:1. To measure progress toward meeting college and careerreadiness standards2. To determine if students are on target for college andcareer readiness3. To provide instructionally actionable information to educators4. To inform evaluation of school and program effectiveness5. To inform readiness for advanced high school courseworkWe documented evidence that Arkansas’ ACT Aspire scores are related as expected with other variables, supporting Critical Element 3.4 for ESSA Peer Review. Additional research is needed to understand how ACT Aspire test scores and high school coursework and grades can be used together for advising and placement. © 2019 by ACT, Inc. All rights reserved | R1762-Rev 1Jeff Allen is a statistician and director in Validity and Efficacy Research. He specializes in longitudinal research of educational outcomes, student growth models, and validation of college readiness measures.Justine Radunzel, PhDJustine Radunzel is a principal research scientist in Validity and Efficacy Research specializing in postsecondary outcomes research and validity evidence for the ACT test.Jizhi LingJizhi Ling is a former data analyst in Validity and Efficacy Research who worked on postsecondary outcomes research.ACKNOWLEDGEMENTSWe thank the Arkansas Department of Education for providing the data necessary to conduct this study. We also thank Michelle Croft, Yu Fang, and Krista Mattern for their feedback on an earlier version of this paper. ContentsIntroduction ................................................................................................................................................. 1 Test-Criterion Relationships: Prediction of High School Course Grades ............................................ 2 Test-Criterion Relationships: Prediction of AP Exam Scores ................................................................ 7 Convergent And Discriminant Evidence: ACT Aspire/PARCC Cross-Grade Correlations ................. 9 Summary .................................................................................................................................................... 12 Relating ACT Aspire Scores to Performance in High School Courses and Other Measures of College and Career Readiness Jeff Allen, PhD, Justine Radunzel, PhD, and Jizhi Ling Introduction In this report, we provide evidence to support Critical Element 3.4 of ESSA Peer Review: “The State has documented adequate validity evidence that the State’s assessment scores are related as expected with other variables” (US Department of Education, 2018). The types of evidence presented in this report are given in Table 1. Evidence is included for each subject area and grade level required for ESSA, as well as the Composite score, which is used as an indicator of career readiness. Evidence is also included for student subgroups: English Language Learner (ELL), Special Education (SPED), free/reduced lunch-eligible (FRL), and three racial/ethnic subgroups–Black, Hispanic, and White. The evidence falls into two categories: test-criterion relationships and convergent and discriminant evidence. We argue that intended interpretations for uses of ACT Aspire test scores imply that the test scores should be related to variables external to the test; thus, predictive (test-criterion) and convergent/discriminant relationships are important sources of validity evidence (AERA, APA, NCME, 2014). Uses of Aspire test scores include (ACT, 2019): 1.To measure progress toward meeting college and career readiness standards2.To determine if students are on target for college and career readiness3.To provide instructionally actionable information to educators4.To inform evaluation of school and program effectiveness5.To inform readiness for advanced high school coursework6.To understand student and group performance relative to national normsIn this report, we document evidence supporting the first five uses. Evidence supporting the sixth use is documented in the ACT Aspire Summative Technical Manual (ACT, 2019).Table 1. Types of Validity Evidence, by Subject Area and Grade