Out of This World

Video by Mike Tate

Survey of Meteorology Concepts: Validation Strategies

Air University Public Affairs

Dec. 7, 2023 | 9:58

Daniel O’Keefe, Louis Cascino, Christopher Francis, Wilson Gonzalez-Espada, & Kimberly de La Harpe

Misconceptions are inaccurate conceptual understandings and cognitive structures rooted in students’ thinking but deviate from the conception described by experts in their field, even after traditional instruction. Misconceptions influence how students receive and process new ideas and can interfere with their ability to form a meaningful understanding of complex concepts. Although researchers have explored misconceptions in several meteorology topics, there is a dearth of comprehensive, valid, and reliable questionnaires that can identify persistent misconceptions. Being well-informed, understanding the nature and impact of hazardous natural events, and implementing appropriate responses is essential for national security. The purpose of this study was to develop the Survey of Meteorology Concepts (SMC) as a literature-based, psychometrically valid, and reliable meteorology questionnaire. Although this is still a work-in-progress, the researchers
would like to discuss the validation strategies, which can be implemented in other surveys and classroom-based assessments. These include (a) Item Response Theory, including reliability coefficients, item difficulty, and item discrimination; (b) Differential Item Functioning, a set of methods that can help identify disparities by sex, ethnicity, age, or education among cadets of the same ability; and (c) Distractor Analysis, a deep-analysis of all the alternatives in a multiple-choice question that, in association with a Certainty of Response Index prompt, can identify to what extent distractors are plausible, and which could represent an actual lack of knowledge of a topic, or a misconception.