Exploring the Relationship Between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive Testing

ACT Researck Report Series 99 =*5Exploring the Relationship Between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive TestingShu-Ying Chen Robert D. Ankenmann Judith A. SpraySeptember 1 9 9 9 For additional copies write:ACT Research Report Series PO Box 168Iowa City, Iowa 52243-0168© 1999 by ACT, Inc. All rights reserved. Exploring the Relationship Between Item Exposure Rate and Test Overlap Rate in Computerized AdaptiveTestingShu-Ying Chen and Robert D. AnkenmannThe University of IowaJudith A. SprayACT, Inc. AbstractThis paper presents a derivation of an average between-test overlap index as a function of the item exposure index, for fixed-length computerized adaptive tests. This relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. Implications for practice as well as future research are also discussed. Exploring the Relationship Between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive TestingThe popularity of computerized adaptive tests (CATs) has increased in recent years due to the significant progress of computer technology. Many conventional paper-and-pencil (P&P) tests, like the Graduate Record Examination (GRE) and the Armed Services Vocational Aptitude Battery (ASVAB), are now offered in a CAT format. One practical advantage of CATs is that they can be administered on a flexible schedule rather than at fixed times. The convenience and flexibility for examinees, however, may severely compromise test security if item exposure is not well controlled. Because test security is always an important concern, especially in high stakes testing programs (e.g., college admissions, or certification and licensure), CATs cannot be implemented effectively in practice unless item exposure is well controlled.Way (1998) stated that, to date, the methods used to avoid item overexposure in CATs fall into two general categories: (a) randomized item selection (e.g., McBride & Martin, 1983; Bergstrom, Lunz, & Gershon, 1992; Way Zara, & Leahy, 1996); and (b) conditional item selection (e.g., Sympson & Hetter, 1985; Davey & Parshall, 1995; Stocking & Lewis, 1995, 1998). Regardless of the item exposure control method used, item exposure rate and average item overlap are two indices commonly used to track item exposure in CATs (Way, 1998). Item exposure rate refers to the relative frequency with which an item is presented across all CAT administrations, that is, the proportion of all CATs in which an item is administered. Average item overlap is defined by Way (1998) as the proportion (or percentage) of item s shared by pairs of exam s, averaged across all possible pairw ise com parisons. It is im portant to note that M ills and Stocking (1996) use the term item overlap in referring to ''the extent to which one item m ay cue the correct response to another item or the extent to w hich two items depend on the same specific know ledge" (p. 294). To avoid confusion, and to provide a m ore accurate and descriptive nom enclature, we introduce the follow ing term inology and definitions: (a) For a pairw ise com parison betw een two fixed-length CA Ts that have been adm inistered, the between-test overlap is the proportion of items on one test that also appear on the other test (i.e., the proportion of shared items); and (b) the average betiveen-test overlap is the arithm etic m ean of the betw een-test overlaps across all possible pairw ise com parisons. Furtherm ore, we use the terms average between-test overlap and test overlap rate interchangeably. The average betw een-test overlap, as defined above, is equivalent to the average item overlap defined by W ay (1998). By considering both the item exposure rate and the average betw een-test overlap, item exposure can be m onitored at the individual item level as well as the test level.D espite the im portance of both item exposure rate and test overlap rate in tracking item exposure control, few studies have investigated the effects of sim ultaneously controlling the m agnitudes of these two indices. W hile m ost research to date has focused on item exposure control at the individual item level, D avey and Parshall (1995) proposed a conditioned item exposure control m ethod designed to function at both the item and test levels. A lthough this m ethod reduces the am ount of test overlap and is m ore general than m ethods that function only at the individual item level, it fails to control the test overlap rate exactly, that is, it fails to ensure desire