The Effect of Item Parameter Estimation Error on Decisions Made Using the Sequential Probability Ratio Test

ACT Research Report Series87-17The Effect of Item Parameter Estimation Error on Decisions Made Using the Sequential Probability Ratio TestResearch Report ONR 87-1Judith A. Spray Mark D. ReckasePrepared under Contract No. N00014-85-C-0241, Contract Authority Identification No. NR 154-531, with the Cognitive Science Research Program of the Office of Naval Research.Approved for public release; distribution unlimited. Reproduction in whole or in part is permitted for any purpose of the United States Government.September 1987 For additional copies write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243©1988 by The American College Testing Program. All rights reserved. UnclassifiedSECURITY CLASSIFICATION OF THIS PAGEREPORT DOCUMENTATION PAGEla. REPORT SECURITY CLASSIFICATION UNCLASSIFIED1b. RESTRICTIVE MARKINGS2a. SECURITY CLASSIFICATION AUTHORITY2b. DECLASSIFICATION/DOWNGRADING SCHEDULEY. DISTRIBUTION /AVAILABILITY OF REPORT Approved for public release: distribution unlimited.Reproduction in whole or in part is permittedf o r a n y p n rp n sp . n f f h e U n i f P H Sfafgg H n v p r nmi4. PERFORMING ORGANIZATION REPORT NUMBER(S) ONR 87-15. MONITORING ORGANIZATION REPORT NUMBER(S)6a. NAME OF PERFORMING ORGANIZATION ACT6b. OFFICE SYMBOL (If applicable)7a. NAME OF MONITORING ORGANIZATIONCOGNITIVE SCIENCE RESEARCH PROGRAMS OFFICE OF NAVAL RESEARCH6c. ADDRESS (C/ty, State, and ZIP Code)P.O. Box 168 Iowa City, IA 522437b. ADDRESS (C/fy, State, and ZIP Code) Code 1142CSArlington, VA 22217-50008a. NAME OF FUNDING/SPONSORING ORGANIZATION8b. OFFICE SYMBOL (If applicable)9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERN00014-85-C-02418c. ADDRESS (City, State, and ZIP Code)10. SOURCE OF FUNDING NUMBERSPROGRAM ELEMENT NO.61153NPROJECTTASKNO.NORR04204RR0420401WORK UNIT ACCESSION NOtNR153-53I11. TITLE (Include Security Classification)The effect of item parameter estimation error on decisions made using the sequentional probability ratio test12. PERSONAL AUTHOR(S)Judith A. Sprav. Mark D. Reckase13a. TYPE OF REPORT Technical13b. TIME COVERED FROM TO14. DATE OF REPORT (Year, Month, Day) 1987. October15. PAGE COUNT 26______16. SUPPLEMENTARY NOTATION17.COSATI CODES18. SUBJECT TERMS (Continue on reverse if necessary and identify by block number)FIELDGROUPSUB-GROUPAdaptive testing Sequential probability ratio tests Item response theory0509Latpnf- trait theorv19. ABSTRACT (Continue on reverse if necessary and identify by block number)A series of computer simulations were performed in order to observe the effects of item response theory (IRT) item parameter estimation error on decisions made using an IRT- based sequential probability ratio test. Specifically, the effects of such error on misclassification rates and the average number of items required for either a mastery (pass) or nonmastery (fail) decision were observed under varied SPRT conditions. These conditions include the a priori or nominal type I (a) and type II (8) error rates, the simple hypotheses tested by the SPRT procedure, and the composition of the item pool (specifically the a, b and c parameters which characterized the items according to a three-parameter logistic model) used to administer the SPRT. The results of these simula­tions showed that these SPRT decisions are not greatly affected by this particular level of error in parameter estimates modeled in this study. Misclassification error rates were slightly greater when estimation error in the item parameters was present, but such differences appear to be negligible.20. DISTRIBUTION/AVAILABILITY OF ABSTRACTE UNCLASSIFIED/UNLIMITED □ SAME AS RPT. □ DTIC USERS21. ABSTRACT SECURITY CLASSIFICATION UNCLASSIFIED22a. NAME OF RESPONSIBLE INDIVIDUAL22b. TELEPHONE (Include Area Code) On?) 696-404622c. OFFICE SYMBOL ONR 1142CSDO FORM 1473,84 m ar83 APR edition may be used until exhausted. All other editions are obsolete.SECURITY CLASSIFICATION OF THIS PAGEUnclassified THE EFFECT OF ITEM PARAMETER ESTIMATION ERROR ON DECISIONS MADE USING THE SEQUENTIAL PROBABILITY RATIO TESTJudith A. Spray Mark D. ReckaseApproved for public release; distribution unlimited. Reproduction in whole or in part ia permitted for any purpose of the United States Government. ABSTRACTA series of computer simulations were performed in order to observe the effects of item response theory (IRT) item parameter estimation error on decisions made using an IRT-based sequential probability ratio test. Specifi­cally, the effects of such error on misclassification rates and the average number of items required for either a mastery (pass) or nonmastery (fail) decisio