The Bootstrap and Other Procedures for Examining the Variability of Estimated Variance Components in Testing Contexts

A C T R esearch R e p o rt Series87-7The Bootstrap and Other Procedures for Examining the Variability of Estimated Variance Components in Testing ContextsRobert L. Brennan Deborah J. Harris Bradley A. HansonSeptember 1987 For additional copies write: ACT Research Report Series P.O. Box 168 Iowa City, Iowa 52243© 1987 by The American College Testing Program. All rights reserved. THE BOOTSTRAP AND OTHER PROCEDURES FOR EXAMINING THE VARIABILITY OF ESTIMATED VARIANCE COMPONENTSIN TESTING CONTEXTSRobert L. Brennan Deborah J. Harris Bradley A. Hanson TABLE OF CONTENTSPageABSTRACT............................................................ iiiTHE p x i RANDOM EFFECTS DESIGN ANDASSOCIATED VARIANCE COMPONENTS.......................................2STANDARD ERRORS AND CONFIDENCE INTERVALSFOR VARIANCE COMPONENTS..............................................5Traditional Approach..............................................6Bootstrap— General Issues.........................................7Bootstrap with the p x i Random Effects Design....................9Jackknife— General Issues.........................................11SIMULATION RESULTS FOR NORMALLY DISTRIBUTED DATA.....................12Data Generation..................................................13Standard Errors..................................................14Confidence Intervals.............................................16Discussion.......................................................18SIMULATION RESULTS FOR BINARY DATA..................................19Bootstrap Sampling Procedures....................................19Standard Errors and Confidence Intervals..........................21SUMMARY AND CONCLUSIONS.............................................24REFERENCES.......................................................... 28APPENDIX A— Estimated Standard Errors and Satterthwaite*sConfidence Intervals for Variance Components......................30APPENDIX B— Jackknife Estimates, Their Standard Errors andConfidence Intervals for the p x i Random Effects Design..........32APPENDIX C— Tables Illustrating Results of Different Procedures for Estimating Variance Components andTheir Standard Errors for Normally Distributed Data..............35FOOTNOTES........................................................... 46TABLES.............................................................. 47 ABSTRACT. This paper examines the applicability of traditional, bootstrap, and jackknife methodologies for estimating standard errors and obtaining confidence intervals for the variance components for persons, items, and residuals in a random effects G study p x i design. Principal consideration is given to simulation results with binary data, although some simulation results for normally distributed data are also reported. The simulations suggest that the traditional approach produces accurate results with normally distributed data but poor results with binary data, at least for the variance component for residuals. The jackknife provides quite accurate results for both types of data and for all three variance components. The bootstrap can be "made to work" reasonably well but doing so seems to require several ad hoc procedures for defining bootstrap samples, which renders the bootstrap somewhat less satisfactory than the jackknife for the application considered here.iii I I I I I I I I I I I I THE BOOTSTRAP AND OTHER PROCEDURES FOR EXAMINING THE VARIABILITY OF ESTIMATED VARIANCE COMPONENTSIN TESTING CONTEXTSThe Standards for Educational and Psychological Testing (APA, 1985) statethat. • . the estimation of clearly labeled components of observed and error score variance is a particularly useful outcome of a reliability study, both for the test developer who wishes to improve the reliability of an instrument and for the user who wants to interpret test scores in particular circumstances with maximum understanding. Reporting standard errors, confidence intervals, or other measures of imprecision of estimates is also helpful, (p. 19)The principal purpose of this paper is to examine the applicability of several methodologies for estimating standard errors and obtaining confidence intervals for variance components in testing contexts. Using the terminology of generalizability theory, the specific context considered here can becharacterized as a G study with a random effects p x i design, in which allexaminees respond to the same set of undifferentiated items.* This design can be used to estimate three basic variance components— one for persons, one for items, and one for residuals.The variability of estimates of these three variance components is examined using traditional, bootstrap, and jackknife methodologies, with principal focus on various bootstrap approaches. Each of these methodologies is described in a subsequent section, which is followed by a discussion of simulation results for normally distributed data and binary data. Binary data