Examining Validity and Coherence in a Cognitively-Based Science Performance Assessment

The purpose of this research was to explore the coherence and effectiveness of an assessment approach that combined principles of cognitive-based assessment, performance assessment, and the Next Generation Science Standards. By drawing on research on learning progressions and cognition in geoscience to design, implement, and analyze an Earth Science performance assessment at the high school level, I explored the challenges and opportunities inherent in a cognitively-based science performance assessment system. The primary research question for this study was: How do cognitively-based performance assessments promote coherence between students’ understanding, responses, and scoring? Four subquestions allowed me to compare observations of student thinking with written responses and scores across multiple modalities in order to characterize the overall coherence of the assessment system. Using a design study approach, an assessment was developed using a two-phase process. First, a construct map was created that outlined a learning progression for each of four geology subdomains: geologic time & stratigraphy; surface processes; plate tectonics; and geologic maps. Second, the construct map guided the development of interconnected performance assessment tasks intended to elicit and measure student thinking within those geology subdomains. Twenty-two high school students engaged in a think-aloud protocol while completing the performance assessment. Student responses from the performance assessment were scored according to a predetermined scoring procedure that generated scores on individual items as well as holistic scores for each construct. Data from student written responses and think-alouds were quantitatively coded in alignment with the cognitive model for the assessment system. I used these data to examine the correlations between student thinking, written responses, and scores, in both item-by-item and holistic modalities. The strength of these correlations varied by construct, but some overall patterns emerged: (1) The design of this cognitively-based science performance assessment was successful in eliciting thinking about all four levels of each construct, and there were instances where student thinking went beyond the intended bounds of specific items. (2) For comparisons of student thinking to written responses or scores, holistic values captured a similar or better level of correlation than individual items, pointing to the important role of holistic scoring in the interpretation phase of this assessment approach. (3) The performance assessment produced scores for three out of four constructs with statistically significant correlations to student thinking. Together, these results show that fully capturing student thinking remains a formidable challenge for the assessment field, but that cognitively-based science performance assessment tasks have significant potential to reveal the extent and breadth of student thinking beyond traditional assessment approaches. The findings in this study have implications for the ways in which different stakeholders in science education, including classroom teachers, curriculum writers, and education leadership, can harness the power of cognitively-based assessment tools to better measure and support student learning.