Unidimensional Approximations for a Computerized Test When the Item Pool and Laten Space are Multidimensional

A C T R esearcli Report Series 9 7 -5Unidimensional Approximations for a Computerized Test When the Item Pool and Latent Space are MultidimensionalJudith A. Spray Abdel-fattah A. Abdel-fattah Chi-Yu Huang C. Allen LauMZT July 1997 For additional copies write;ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 1997 by ACT, Inc. All rights reserved. Unidimensional Approximations for a Computerized Classification Test When the Item Pool and Latent Space Are MultidimensionalJudith A. Spray Abdel-fattah A. Abdel-fattah Chi-Yu Huang ACTC. Allen Lau The Psychological Corporation ABSTRACTThe primary concern or focus of a certification or licensure test is to obtain valid criterion-referenced information regarding a candidate's competency to practice. When the test is administered by computer, a valid pass/fail decision can be made with fewer items than an equivalent paper/pencil test by targeting items at the passing score and using a likelihood ratio approach such as the one utilized in the sequential probability ratio test or SPRT. When administered on a computer, the SPRT is frequently referred to as a computerized classification test or CCT (to distinguish it from the usual computerized adaptive test or CAT). If the CCT is IRT-based, an assumption of unidimensionality is usually required, and the concern is when the item pool is not essentially unidimensional. This study investigated the effects that a multidimensional item pool and latent ability space have on the accuracy of the decisions made using CCT The results show that the procedure may be fairly robust to such assumption violations. Un id im e n s io n a l Ap p r o x im a t io n s f o r a Co m p u t e r iz e d Cl a s s if ic a t io n Te s t Wh e n t h e It e m Po o l a n d La t e n t Sp a c e Ar e Mu l t id im e n s io n a l1The primary concern or focus of a certification or licensure test is to obtain valid criterion-referenced information regarding a candidate's competency to practice. When the test is administered by computer, a valid pass/fail decision can be made with fewer items than an equivalent paper/pencil test by targeting items at the passing score and using a likelihood ratio approach such as the one utilized in the sequential probability ratio test or SPRT. When administered on a computer, the SPRT is frequently referred to as a computerized classification test or CCT (to distinguish it from the usual computerized adaptive test or CAT) and a high degree of accuracy of the classification decision can be obtained (Spray & Reckase, 1996). The concern is when the item pool is not essentially unidimensional, because many of the professional certification and licensure examinations are constructed from complex blueprints with content domains, cognitive levels, and practice levels as typical blueprint dimensions. Even when the blueprint consists only of different content categories, they are frequently quite diverse. For example, it is common to see a professional practice blueprint contain very specific categories covering the professional subject matter as well as more general areas such as Professional Issues or Administration.When we first began to study the possible effects of a multidimensional item pool on the classification accuracy of CCT using SPRT procedures, we anticipated that we would be able to produce a multidimensional SPRT procedure analogous to the unidimensional one, so that, if it was determined that an item pool was, indeed 2multidimensional, we could implement the modified procedure to guarantee minimum classification errors. Having access to multidimensional IRT estimation procedures ensured that we could calibrate a multidimensional item pool and then simply implement the modified SPRT CCT algorithms designed to handle the multidimensionality of the item pool.The Unidimensional CaseIn the unidimensional case, assume that the item pool fits say, the 3-PL IRT model, P(Y=1 |0,fl,f7,c). A passing score is established by using some subset of the item pool called a standard reference set. The standard reference set of m items within the item pool mirrors the examination blueprint in terms of major category definitions, proportions of items included within each domain, item difficulty, and other pertinent characteristics. A passing score or passing rate, p, is obtained on the standard reference set by some established method (e.g., the Angoff procedure) and an equivalent latent passing score is obtained in the usual way by solving for 0 in the relationship,1 mp = —EPi(Yi=1le/«AcP' wm ,=1The value of 0 that satisfies this relationship,