Estimating Conditional Probabilities of Success and Other Course Placement Validity Statistics Under Soft Truncation

ACT Researcli Report Series 9Estimating Conditional Probabilities of Success and Other Course Placement Validity Statistics Under Soft TruncationJeff SchieiApril 1998 For additional copies write:ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 1998 by ACT, Inc. All rights reserved. Estimating Conditional Probabilities of Success and Other Course Placement Validity Statistics Under Soft TruncationJeff Schiel Table of ContentsAbstract................................................................................................................................................iiiAcknowledgments....................................................................................................................... ivData ............................................................................................................................................... 6M ethod.......................................................................................................................................... 9Results .......................................................................................................................................... 14Estimated Conditional Probabilities of Success .................................................... 14Estimated Accuracy Rates and Optimal Cutoff Scores....................................... 24Estimated Success Rates .............................................................................................. 36Discussion...................................................................................................................................... 36References...................................................................................................................................... 39Appendix AJoint Distributions of ACT Score andCollege Course Outcome for Placement Groups 2-5 ...................................................... 40Appendix BEffects of Soft Truncation onEstimated Success Rate, by Placement Group ................................................................... 46ii AbstractACT's Course Placement Service uses logistic regression to model the relationships between outcomes in standard college courses and placement test scores. The logistic regression results are used together with empirical test score data to obtain estimates of validity statistics (e.g., proportion of correct placement decisions), given particular hypothetical cutoff scores. Prior research has shown that the accuracy of these statistics is affected by hard truncation, a condition in which no standard course outcome data are available for students below a certain cutoff score. Hard truncation is uncommon in actual placement systems, however; standard course outcome data typically are available for some students below the cutoff score (soft truncation). A soft truncation condition of 40%, for example, is one in which 40% of the students below the cutoff score do not have standard course outcome data. The results of this study show that reasonably accurate estimates of validity statistics can be obtained even when 40%, and in some cases 60% or 80%, soft truncation occurs. Moreover, the slope of the logistic regression curve and the skewness of the test score marginal distribution have little to do with the relative accuracy of validity statistics unless soft truncation exceeds 40%. The efforts of ACT Summer Intern Shireen Pinheiro, who spent many hours testing prototypes of programs, are especially appreciated. The author also thanks the following colleagues:® Jill Crouse, for providing access to the simulated data sets examined in thisstudy;© Chuan-Ju Lin, for helping with the somewhat tedious task of generating12,000 truncation samples; and ° Julie Noble and Richard Sawyer, for offering helpful suggestions for thedesign of the study.Acknowledgmentsiv Estimating Conditional Probabilities of Success and Other Course Placement Validity Statistics Under Soft TruncationPostsecondary institutions often use standardized test scores for deciding how to place students into college-level courses. Typically, students scoring at or above a particular cutoff score are placed into a standard course, whereas those scoring below the cutoff score are placed into a lower-level course, sometimes referred to as a ’’remedial" course. It is very important that correct placement decisions be made; otherwise, students may be adversely affected. For example, entering freshmen who are incorrectly placed into a standard course may be unable to complete it with a passing grade because they lack sufficient academic knowledge and skills. This situation could, of course, be disheartening for students.Placement decisions may also affect an institution