Estimating Item Parameters from Classical Indices for Item Pool Development with a Computerized Classification Test

esearcli R ep o rt S eries 2 0 0 0 -4Estimating Item Parameters from Classical Indices for Item Pool Development with a Computerized Classification TestChi-Yu Huang John C. Kalohn Chuan-Ju Lin Judith SprayJO TJVLarcJi 2000 For additional copies write:ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 2000 by ACT, Inc. All rights reserved. Estimating Item Parameters from Classical Indices for Item Pool Development with a Computerized Classification TestChi-Yu Huang John C. Kalohn Chuan-Ju Lin Judith Spray AbstractItem pools supporting computer-based tests are not always completely calibrated. Occasionally, only a small subset of the items in the pool may have actual calibrations, while the remainder of the items may only have classical item statistics, (e.g., p-values, point-biserial correlation coefficients, or biserial correlation coefficients). Transformations can be applied to the classical statistics to obtain rough estimates of the item parameters from a 3-parameter logistic IRT model. These estimates, in turn, can be improved by linking them to items with actual calibrations from a program such as BILOG. The resulting item-parameter estimates can then be used in a computerized classification test (CCT). An evaluation of the results of using such estimated parameters in simulated CCTs is presented in this paper. Estimating Item Parameters from Classical Indices for Item Pool Development with aComputerized Classification Test1M oving a testing program from paper/pencil to com puterized testing may require that an item pool replace some set of fixed test forms. For many types of com puter-based tests (CBTs), an item pool that has been calibrated and scaled to a latent metric is desired. In practice, however, having a com plete set of item responses for calibration purposes on all items in the pool may be an unreachable goal for some testing programs. Only one or two recently adm inistered paper-pencil test forms might be calibrated and the rest of the item pool may just consist of classical item parameters such as p-values and biserial correlation coefficients for each single item. If these testing programs only require a simple classification decision to be made (e.g., pass/fail), it may be possible to use some m ethods of approxim ation when calibrating the item pool and still achieve valid classification results. The purpose of this paper is to describe a procedure which links IRT-calibrated items based on a small portion of an item pool to the rem ainder of a classically based item pool. The m ajor research question of this study was, “Do these pseudo-calibrations perform as well as actual IRT calibrations obtained from programs such as BILOG in one particular CBT application, nam ely that of a com puterized classification test (CCT)?”1 Portions of this paper were presented at the 1999 annual meeting o f the Psychometric Society in Lawrence, KS. The co-authors of the paper are listed alphabetically. Description of the ProblemAssume that a 360-item pool for a computerized classification test (CCT) consists of 60 calibrated items from one previously administered paper/pencil test. This set of items will be referred to as the standard reference set or SRS. The remainder of the items in the item pool possess their classical item statistics, p-values and either point-biserial or biserial correlation coefficients (pPbS and pbS, respectively, or abbreviated as r and R). The research question to be answered is, “Can item-parameter estimates be obtained on the 300 items that only have classical statistics, and then can these estimates, along with the calibrated SRS, be used to administer a CCT using the sequential probability ratio test (or SPRT) method?” Because the methods described by Urry (1974) and Schmidt (1977) were used to transform an item’s classical statistics into estimates of the a- and /^-parameters from the three parameter logistic model (3- PLM), it is helpful to review those procedures.The Urry-Schmidt TransformationsUrry (1974) proposed that the transformations first described by Bimbaum in Lord and Novick (1968, chapter 16), be corrected for guessing by incorporating a lower bound for the probability of a correct response. Schmidt (1977) refined this method by adjusting for the unreliability of the estimate of the latent trait, 0, in the estimation of R.Under the assumption that 0 ~N(0,1), and the free response (i.e., no guessing) items on a test of length n measure the unidimensional trait, 0, and the response functions of those items can each be described by the usual normal ogive response function, P, orY,(