This paper was the report of a working group which convened as part of the ITiCSE 2004 conference (i.e. the conference on “<I>Innovation and Technology in Computer Science Education</I>). The ITiCSE 2004 conference was held in Leeds, and as there was only one working group that year, this working group is known as “The Leeds Group”. Three years earlier, at ITiCSE 2001, the “McCracken Group” (so named after its leader, Mike McCracken) had assessed the programming ability of a large population of students&nbsp; from&nbsp; four&nbsp; universities in two countries,&nbsp; the&nbsp; United&nbsp; States and&nbsp; the United Kingdom.&nbsp; The students were asked&nbsp; to solve a given programming task.&nbsp;&nbsp;The majority of students performed much more poorly than expected.&nbsp; The McCracken group concluded that the students were weak at problem-solving. </P>

<P>The starting point for the Leeds Group was to question the conclusion drawn by the Leeds Group – that the students performed poorly because of a weakness in problem solving. There were other, simpler explanations that were not eliminated by the experimental design of the McCracken group. For example, perhaps the students did not have a strong grasp of the programming constructs they needed to use to produce a solution? &nbsp; The Leeds Group looked for simpler explanations of the McCracken Group’s result by testing over 500 students (from seven countries) on their ability to read and comprehend code. The data for the Leeds group was collected by asking students to answer twelve Multiple Choice Questions (MCQs). These twelve MCQs were of two types. One type of question tested students on their ability to select, from the four options given, the outcome of executing a short piece of code. For the other type of question, students were given pieces of code with one or two lines missing. The students were told what the complete code should do, and their task was to select the correct missing line(s) from the four options given. Most students performed poorly on these two types of tasks, indicating that the conclusion of the McCracken Group (i.e. that students were weak at problem-solving) may not be the simplest explanation of the McCracken Group’s results. </P>

<P>Apart from recording how many questions each student answered correctly, the Leeds Group also collected two other types of data.&nbsp; One of these other types of data consisted of transcripts. Thirty seven students were asked to “think out loud” as they answered the twelve MCQs. These students were recorded, and the tapes were transcribed.&nbsp; The other type of data was referred to as “doodles”. &nbsp;&nbsp;This data was the annotations students made as they attempted to answer these questions.&nbsp; Both of these forms of data showed that students used simplistic approaches to answering the questions; approaches that would not equip them to write code of similar complexity.&nbsp; </P>

<P>[1] McCracken,&nbsp; M.,&nbsp; V.&nbsp; Almstrum,&nbsp; D.&nbsp; Diaz,&nbsp; M.&nbsp; Guzdial,&nbsp; D. Hagen,&nbsp; Y.&nbsp; Kolikant,&nbsp; C.&nbsp; Laxer,&nbsp; L.&nbsp; Thomas,&nbsp; I.&nbsp; Utting, T.&nbsp;&nbsp;&nbsp; Wilusz,&nbsp;&nbsp;&nbsp; (2001)&nbsp;&nbsp;&nbsp; A&nbsp;&nbsp;&nbsp; Multi-National,&nbsp;&nbsp;&nbsp; Multi Institutional&nbsp; Study&nbsp; of&nbsp; Assessment&nbsp; of&nbsp; Programming Skills&nbsp; of&nbsp; First-year&nbsp; CS&nbsp; Students,&nbsp; SIGCSE&nbsp; Bulletin,&nbsp; 33(4). pp 125-140.</P>

Author:<br>