The current trend in K-12 STEM education is to implement engineering design-based instruction as a pedagogical approach to teaching STEM concepts. Specific evidence of this trend the movement of over fifteen states beginning to implement engineering and technology focused standards at the K-8 level, including strands such as The Design Process (Indiana Department of Education, 2011), The Nature of Science and Engineering (Minnesota Department of Education, 2009), and The Nature of Technology/Engineering (Massachusetts Department of Education, 2006). Another example is the proposed Conceptual Frameworks for New Science Education Standards (NAS, 2012) that includes engineering standards as an integral part of the framework. This trend is similar to reform in engineering education within the last two decades. Atman and Bursic (1998) indicated at during that time of engineering education curriculum reform there was an increased focus on design, thus, a need existed to locate assessment practices that allowed researchers to move beyond assessing design products to assessing the process taken by designers. Atman and Bursic chose the think-aloud protocol as a research methodology to assess the design process taken by undergraduate engineering majors. Today with a barrage of K-12 STEM education curriculum efforts (Project Lead the Way, Infinity Project, Engineering is Elementary, Engineering by Design, Children Designing and Engineering), it is important to once again locate assessment practices that shed light on the process taken by students as they learn STEM concepts through a design based approach to learning. Researchers on the NSF targeted MSP Science Learning through Engineering Design (SLED) have employed a transfer problem and think-aloud protocol analysis (Atman & Bursic, 1998; Erecsson & Simon, 1993) to assess students‟ transfer of learning from classroom engineering design-based experiences. Although this assessment process is limited to small purposeful sample sizes, the technique allows a researcher to observe in real time students (teams of three) engaged in authentic design thinking and identify which science concepts naturally emerge in the design team‟s dialogue. The researcher is able to assess the accuracy of the science concepts employed and identify any misconceptions that remain. Using a special coding process (Halfin, 1973), SLED researchers have successfully piloted a coding scheme for different cognitive strategies exhibited by elementary school students. The SLED researchers also have systematically identified gaps and areas of overemphasis of the design process. The coding results were organized using frequency and percentages for time on code. Preliminary analysis of data collected from pilot studies within the SLED project indicated limited use of science terms within design thinking dialogue; however the science terms that did emerge were used accurately and appropriately. Preliminary findings of early data collection for the SLED project will be presented along with discussion regarding limitations of this methodology as an approach to assessing student learn.

Todd Kelley

A paper presentation at the P-12 Engineering and Design Education Summit

Researchers should cite this work as follows:

• Todd Richard Kelley (2012), "Assessment on Student Learning Design: Concurrent Think-aloud Protocols," https://stemedhub.org/resources/969.

  BibTex | EndNote

1. Design Research
2. Research paper
3. Think aloud Protocols