Table 3 Assessment of test items using the preliminary conceptual framework for establishing content validity of AI-generated test items (integrated case clusters, SAQs and OSPEs)
Types of errors | Sage Poe© | Chart GPT© | Claude-Instant© | |||
|---|---|---|---|---|---|---|
Pre-clerkship | At graduation | Pre-clerkship | At graduation | Pre-clerkship | At graduation | |
Integrated case cluster | ||||||
Technical accuracy | Complete | Complete | Complete | Complete | Deficient | Complete a |
Comprehensiveness | Deficient | Deficient | Deficient | Deficient | Deficient | Deficient a |
Education level | Complete | Deficient | Deficient | Deficient | Complete | Complete a |
Free of construction defects | Deficient | Deficient | Deficient | Deficient | Deficient | Complete a |
Short answer questions | ||||||
Technical accuracy | Complete | Complete | Complete | Complete | Complete | Complete |
Comprehensiveness | Complete | Complete | Complete | Complete | Complete | Complete |
Education level | Complete | Deficient | Complete | Deficient | Complete | Complete |
Free of construction defects | Deficient | Deficient | Deficient | Deficient | Deficient | Complete |
OSPEs | ||||||
Technical accuracy | Not Available | Complete | Complete | Complete | Complete | |
Comprehensiveness | Complete | Complete | Complete | Complete | ||
Education level | Complete | Complete | Complete | Complete | ||
Free of construction defects | Complete | Complete | Complete | Complete | ||