Validating Two-Stage Course Placement Systems When Data Are Truncated

h R e p o rt©searcin axcbofc oenesValidating Two-Stage Course Placement Systems When Data Are TruncatedJeff L. Schiel Matt HarmstonJjO'TF F e b ru a ry 2 0 0 0 For additional copies write;ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 2000 by ACT, Inc. All rights reserved. Validating Two-Stage Course Placement Systems When Data Are TruncatedJeff L. Schiel Matt Harmston AbstractIn two-stage course placement systems, students first take a screening test. Students who score at or above the screening test cutoff .score K enroll directly in a standard college course, whereas those who score below K take a placement test. Students who subsequently score at or above the placement test cutoff K' also enroll in the standard course. Consequently, students in the standard course will not have placement test scores below K'. Moreover, placement test scores are somewhat truncated above K\ because students who earned scores above K on the screening test did not have to take the placement test. Hence, their placement test scores, which likely would have equaled or exceeded K', are “missing.”Previous research has only examined truncation in one-stage placement systems, in which it occurs below, but not above, the cutoff score. In this study, the effccts of truncation on estimated optimal cutoffs, accuracy rates, and success rates under different combinations of logistic regression curve, test score distribution shape, and sample size were examined for two-stage placement systems. It is shown that even when data are moderately truncated in such systems (e.g., baseline truncation below K' and 80% truncation above K'), validity statistics and optimal cutoffs can be estimated with reasonable accuracy. Validating Two-Stage Course Placement Systems When Data Are TruncatedPostsecondary institutions often use standardized test scores when deciding into which courses students ought to be placed. After selecting a cutoff score, institutional staff will permit students scoring at or above it to be placed into a standard course (e.g., pre-calculus). Students scoring below the cutoff will be placed into a lower-level, remedial course (e.g., college algebra). For the benefit of their institutions and students, institutional staff want to make correct placement decisions, of which there arc two types: i) students placed into a standard course have the necessary skills and knowledge to ultimately succeed in the course, and 2) students placed into a remedial course would not have succeeded in the standard course had they instead been placed into it. Incorrect placement decisions may negatively affect both students and institutions. For example, a student with better-than-average mathematical skills who is incorrectly placed into a remedial mathematics course may become frustrated by the expense and time required to complete an additional course, and may consider transferring to another institution.If students, parents, or others perceive placement systems as being unfair or hastily developed, then these systems may be criticized. By establishing statistical validity evidence that relates standardized test scores or other variables to successful performance in standard courses, institutions can strengthen their respective rationales for using certain placement procedures, tests, and cutoff scores. In this way, institutions are better prepared to respond to potential criticism of their placement systems.One method for providing course placement validity evidence uses logistic regression and decision theory to describe relationships between outcomes in standard college courses and test scores, estimate proportions of correct decisions given particular cutoff scores, and identify optimal cutoffs (ACT, 1994; Noble & Sawyer, 1997; Sawyer, 1989; Sawyer, 1996). In evaluating course placement systems, logistic regression can beused to estimate the conditional probability of success P in a standard course, given test score (or other predictor variables). Estimated probabilities can then be used with the marginal distribution of test scores to estimate other course placement validity statistics,such as the accuracy rate A, which is the estimated proportion of correct placementdecisions. The optimal cutoff score is the cutoff score at which A is maximized.Another validity statistic, the success rate S , is the estimated proportion of students succeeding in the standard course, among all students who could have been placed in that course.Because students who score below the cutoff typically do not enroll in the standard course and do not