The Effects of Data Truncation on Estimated Validity Indices for Course Placement

ACT Research Report Series92-3The Effects of Data Truncation on Estimated Validity Indices for Course PlacementJeff Schlel Julie IMobleOctober 1992 For additional copies write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243© 1992 by The American College Testing Program. All rights reserved. THE EFFECTS OF DATA TRUNCATION ON ESTIMATED VALIDITY INDICES FOR COURSE PLACEMENTJeff Schiel Julie Noble ABSTRACTTraditionally, correlation coefficients have been used to validate course placement decisions based on test scores and high school grades. Because placement systems restrict the range of both the predictor and outcome variables, correlation coefficients based on data from students enrolled in particular courses are understated relative to what they would be if placement had not occurred. Alternative methods have therefore been examined for validating placement systems. One such approach uses validity indices estimated from logistic regression analyses and distributions of predictor variables to determine placement effectiveness.The ASSET Basic Skills test scores and course grades of entering freshman from four postsecondary institutions were analyzed to determine the impact of prior selection on the accuracy of estimated validity indices. Estimated validity indices based on truncated distributions of test scores and course grades were compared to the same indices based on full distributions. It was found that greater degrees of truncation are associated with a loss of accuracy in estimated validity indices. However, the loss of accuracy in the estimates was small when less than 15% of the data for the full distributions were truncated. THE EFFECTS OF DATA TRUNCATION ON ESTIMATED VALIDITY INDICES FOR COURSE PLACEMENTIt is common practice for postsecondary institutions to use standardized test scores for placing students into college-level courses. If a student's test score is at or above a specified cutoff, then she or he would be placed into a standard-level course. If instead the student's score is below the cutoff, she or he would be placed into a developmental or lower-level course.Placement decisions, whether correct or incorrect, may affect individual students in several ways. For example, if a student is incorrectly placed in a standard-level course, she or he may be unable to complete it satisfactorily because the level of the course work exceeds the student's level of knowledge and skills. On the other hand, if a student is placed in a developmental course, then she or he may have to pay additional tuition, simply because of the extra course work that must be undertaken. Further, the student may have to allocate more time toward earning a degree than she or he originally anticipated. If the student is incorrectly placed in the developmental course, then the level of course work may not be sufficiently challenging, and she or he may become discouraged.Placement decisions may also affect the institution. If many students are identified as needing remediation, for example, it may be necessary to schedule extra sections of a particular developmental course or to hire additional teaching staff. If the students are incorrectly identified as needing remediation, such hiring or scheduling efforts may be superfluous.Because of the importance of placement decisions, it is essential that they be as accurate as possible. If test scores are used to make these decisions, but are not valid for use in course placement, then placement decisions based on the scores cannot be accurate. Traditionally, correlation coefficients have been used to document the strength of the statistical relationships between test scores and course grades, and thereby serve as a measure of the validity of the test scores. There are, however, some disadvantages associated with using correlation coefficients for this purpose.At most institutions, students are placed into standard-level courses using test scores and/or other related information. Students scoring above a specified cutoff score are placed into the course, while students scoring below the cutoff are placed into remedial courses. When outcomes (i.e., grades) for the standard course are examined and associated with test scores, correlations between test scores and course grades can only be developed for students placed in the standard-level course. Thus, due to prior placement, the range of the test scores is restricted. Moreover, if the placement test effectively identifies high-risk students, there will be few students in the standard course who earn poor grades; therefore, the range of course grades will also be restricted. The magnitude of