Accuracy of Course Placement Validity Statistics Under Various Soft Truncation Conditions

A C T Rese& rcli R e p o rt S eriesAccuracy of Course Placement Validity Statistics Under Various Soft Truncation ConditionsJeff L. Schiel Jason E. KingK?TJanuary 1999 For additional copies write:ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 1999 by ACT, Inc. All rights reserved. Accuracy of Course Placement Validity Statistics Under Various Soft Truncation ConditionsJeff L. SchielJason E. King L AbstractAnalyses of data from operational course placement systems are subject to the effects of truncation: Students with low placement test scores may enroll in a remedial course, rather than a standard-level course, and therefore will not have outcome data from the standard course. In “soft truncation,” some (but not all) students who score below the cutoff for a standard course enroll in and complete the course. Previous research, using one particular definition of soft truncation, showed that reasonably accurate validity statistics can be estimated under this condition. Alternative definitions of soft truncation could conceivably result in different validity statistics. This simulation study therefore examined an alternative definition of soft truncation, in which students who score just below the cutoff have a higher probability of enrolling in the standard course than do relatively lower-scoring students.The effects of different combinations of soft truncation condition, logistic regression curve, test score distribution shape, and sample size on estimated optimal cutoff scores, accuracy rates, and success rates were summarized. Postsecondary institutions that experience a moderate degree of soft truncation (e.g., 20% to 60% of their respective placement groups) can expect to obtain acceptably accurate estimates of optimal cutoff scores, irrespective of the steepness of the logistic curve and the skewness of the marginal distribution of the predictor variable. Accuracy of Course Placement Validity Statistics Under Various Soft Truncation ConditionsIt is common practice for postsecondary institutions to use standardized test scores for placing students into courses. Of particular interest to these institutions is establishing cutoff scores that will result in a high percentage of correct placement decisions: Students scoring at or above a particular cutoff score are placed into a standard course and are ultimately successful in it; those scoring below the cutoff are placed into a remedial course and would not have succeeded in the standard course had they been placed into it. Incorrect placement decisions are likely to have negative consequences for both students and institutions. The student who is incorrectly placed into standard freshman English, for example, but lacks the required skills and knowledge to complete the course with a passing grade may become discouraged about his or her academic progress.It is important for postsecondary institutions to establish statistical validity evidence for using test scores in course placement. Such evidence provides a rationale for using particular tests, other variables, and cutoff scores. The institution can then use the evidence to respond to potential criticism of its placement practices.Logistic regression and decision theory are well suited for describing relationships between outcomes in standard college courses and test scores, for establishing cutoff scores, and for providing course placement validity evidence. With logistic regression, a binary outcome variable (e.g., course success or failure) can be modeled as a function of test score,yielding an estimated conditional probability of success (P) in the standard course. Estimated conditional probabilities obtained from a logistic regression model can then be used with the marginal distribution of test scores to estimate other course placement validity statistics, such as accuracy rates and success rates, which in turn can be used to identify an optimal cutoff score. The optimal cutoff score is the cutoff score at which the estimatedaccuracy rate (A) is maximized.In the context of evaluating course placement systems, A is defined as the proportion of correct placement decisions, and focuses on estimated probabilities of success for two groups of students: 1) those students scoring at or above the cutoff score for a standard course who are adequately prepared for and successful in the course, and 2) those scoring below the cutoff score who need remedial instruction and therefore would not have been successful inthe standard course had they enrolled in it. The estimated success rate (S) is defined as the proportion of students succeeding in the standard course, among all students who could have