Effects of Item-Selection Criteria on Classification Testing with the Sequential Probability Ratio Test

A C T R esearcla R eport Series 2 0 0 0Effects of Item-Selection Criteria on Classification Testing with the Sequential Probability Ratio TestChuan-Ju Lin Judith SprayJICT Jnine For additional copies write:ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 2000 by ACT, Inc. All rights reserved. Effects of Item-Selection Criteria on Classification Testing with the Sequential Probability Ratio TestChuan-Ju Lin Judith Spray ABSTRACTThis paper presents comparisons among three item-selection criteria for the sequential probability ratio test. The criteria were compared in terms of their efficiency in selecting items, as indicated by average test length (ATL) and the percentage of correct decisions (PCD). The item-selection criteria applied in this study were the Fisher information function, the Kullback- Leibler information function, and a weighted log-odds ratio. We also examined the effects of the cutoff scores, the width of the indifference region, the item pool size, and the item exposure rate under the different item-selection criteria. The results of the computer simulations showed that the three criteria yielded very small differences in the outcome measures, regardless of the conditions imposed. 1EFFECTS OF ITEM-SELECTION CRITERIA ON CLASSIFICATION TESTING WITH THE SEQUENTIAL PROBABILITY RATIO TEST IntroductionComputerized adaptive testing (CAT) is receiving more attention and has been applied more commonly over the last few years. Adaptive testing can yield more efficient tests by saving testing time (i.e., shorter tests) and increasing measurement precision. If the purpose of a test is to classify examinees into one of two or more mutually exclusive categories rather than estimating ability levels, the CAT procedure can be applied to make efficient decisions of classification by selecting and administering optimal items with algorithms based on statistical hypothesis testing, such as the sequential probability ratio test or SPRT (Spray & Reckase, 1994,1996). The main purpose of this study was to compare three item-selection criteria in terms of average test length (ATL) and percentages of correct decisions (PCD) in the context of item selection with the SPRT. Variables hypothesized to affect ATL and PCD included the choice of the item-selection criteria, position of cutting points on the ability metric, the width of the indifference region, item pool size, and item exposure rate. Three types of selection criteria, three different cutting points, 11 indifference regions, two different item pool sizes, and three item exposure rates were examined.The SPRTWald’s (1947) SPRT has been applied for classifying examinees into two mutually exclusive categories using a computerized adaptive test (Eggen, 1999; Spray & Reckase, 1996). In order to distinguish the computerized SPRT from conventional CAT, the SPRT is usually regarded as a computerized classification test or CCT (Spray, Abdel-fattah, Huang, & Lau, 1997). In criterion-referenced testing situations, it is necessary to decide between two hypotheses, Hi and H2, which can be written arbitrarily asH i: 0 < 0o - 5 = 0i vs.H2:0>0o + 8 = 02,where 0 represents the ability of an examinee, 0o is a given cutting point or passing criterion, 0i and 02 refer to the lower and upper bounds, respectively (i.e., we assume that 02 > 0i), of a particular decision threshold, and where 5 forms a small region, called an indifference region, on both sides of the cutting point. The width of the indifference region or interval of 02 - 0i usually equals 26.1 Two decision error rates, a (i.e., type I error rate or false positive) and (3 (i.e., type II error rate or false negative) can be defined as follows: P(choosing H2I Hi is true) = a vs. P(choosing Hi I H2 is true) = |J. The test statistic used in SPRT is a likelihood ratio, which is a ratio of the likelihood functions under the alternative (H2) and null hypotheses (Hi), ornp,(e2)x-[i-p,(02)]1"'LR(x) = = i=l------------= M-------------------------------, (1)“ Z.(e,;AT) * * , r1 n ( 0. ) d - p,(e!>]1=1 i=lwhere L denotes the likelihood function, k represents the number of items or the test length, x contains observed dichotomous item responses, xi, x2, . Xj,.. ,xk, andp,(0i) and p,(02) define the probabilities of a correct response to item i, conditional on 0i and 02. Equation (1) indicates that the higher the ratio, the more likely an examinee would be above the cutting point; the smaller the ratio, the more likely an examinee would be below the cutting point. According to Wald (1947), the nominal error rates, a and P, can be determined before test administration because2