Multiple-Category Classification Using a Sequential Probability Ratio Test

ACT Research Report Series93-7Multiple-Category Classification Using a Sequential Probability Ratio TestJudith SprayDecember 1993 For additional copies write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243©1993 by The American College Testing Program. All rights reserved. Multiple-Category Classification Using a Sequential Probability Ratio TestJudith A. Spray American College Testing ERRATA - Please replace page three of Research Report 93-7 with this page 3variable, Xy represents a single Bernoulli trial and is distributed as Bin{P(0j),l}. Then,B-(0i) = Prob(X = x |6 = 0j) = P ^ )* QCej1-*, where1, correct responsex -0, incorrect responseFor this test item, the probability of observing X = x under the alternative hypothesis is U nder the null hypothesis, the probability of observing X = x is tt(0o). The functions, andtt(0o), are called likelihood functions of jc, and a ratio of these two functions, L(x) = 7r(01)/7r(0O),is called a likelihood ratio.Two error probabilities, a and 0, can be defined, whereProb(choosing H : if H 0 is true) = a andProb(choosing H0 if H x is true) = /?.W ald (1947) stated that even though the nominal error rates, a and f3, are established prior to testing, the actual error rates observed in practice, a* and /?’, are bounded from above by functions of the nominal rates, or a* < a / ( l-(3) and < >3/(1 -a). Wald (1947) also defined two likelihood ratio boundaries that are functions of a and j8. These boundaries are A and B, where the lower boundary = B > 0 /(l-a ) and the upper boundary = A < (1 -(3)/a.According to W ald’s SPRT, item responses are observed in sequence, xv x2, *n, andfollowing each observation, the likelihood ratio, L(xt, x2^100,0^, is computed, assuming conditional independence, where^ ,(0,) w2(0,) ... *rn(e,)L(x„ x2, ...,Jfn|0 o,0l) = ------------------------------------ .3 AbstractSequential probability ratio testing (SPRT), which usually is applied in situations requiring a decision between two simple hypotheses or a single decision point, is extended to include situations involving k decision points and [(£ + l)-choose-2] sets of simultaneous, simple hypotheses, where k> \ . The multiple-decision point or multiple-category SPRT procedure can be used to classify examinees into k + 1 categories using computer adaptive methods. Computer simulations utilizing a 2 0 0-item pool of previously calibrated test items show that the multiple- category SPRT method controls misclassification error rates adequately, provided that the number of decision points is not too large.1 Multiple-Category Classification Using a Sequential Probability Ratio TestW ald’s (1947) sequential probability ratio testing (SPRT) procedure has been used with cognitive tests to classify examinees into one of two categories (e.g., pass/fail, master/nonmaster, certified/noncertified) (Reckase, 1983). In other words this procedure is useful for determining whether an examinee more likely belongs to one of two states or conditions: either an individual has ability or latent trait greater than or equal to some minimum value, 5 or that same individual has ability less than the minimum value, 5. The value, 5, is frequently called a passing score or decision point.One way to test the composite hypothesis that either the examinee has latent ability less than 5 versus that the examinee has latent ability greater than or equal to 6, is to consider simple hypotheses, H0 or H,, regarding the unidimensional latent trait or ability (0S) of the examinee taking the test. These simple hypotheses can be written asH0: 0j = 0O vs.H,: Qi = 6, ,where 0; is an unknown parameter of the distribution of the dichotomous response to a particular test item, X (Silvey, 1975). Usually, 0O and 0, represent decision points that correspond to lower and upper limits, respectively, of the passing criterion or threshold, 8, where 0O < 5 < 0,. The SPRT can then be used to test the composite hypotheses, H0: 0j < 5 versus Hj: 0j > 5 by considering two weaker hypotheses, say co0 = {0:0<0O} and C0j = {0:0>0j} (Silvey, 1975; Wald, 1947).In the case of cognitive testing, X can be assumed to follow a binomial distribution. If P(0j) is the probability that examinee i responds correctly to an item, and Q(0j) = 1 - P(0j) is the probability of an incorrect response from examinee /, then, for this single item, the random variable, X, represents a single Bernoulli trial and is distributed as Bin{P(0j), 1}. Then,71(0,) = Prob(x = *|e = e,) = P(e,r QW1,where1, correct response x =0, incorrect response .For