Results from the Evaluation of the Effectiveness of an Online Tutor on Expression Evaluation
Context of the study: We have been developing online tutors called problets (www.problets.org) to help students learn computer programming concepts by solving problems. This study was conducted to evaluate problets and their features.
Research questions investigated: Are online tutors effective? Do they help students learn? What features contribute to the effectiveness of tutors for Computer Science? Answers to these questions are of interest to developers and users of online tutors in engineering education.
Methodology, Data Collection and Analysis Techniques: An online tutor on arithmetic and relational expression evaluation was used in multiple sections of Computer Science I course from spring 2001 through fall 2002 at the host institution. Controlled tests were used to test a different hypothesis each semester. The traditional pre-test, practice, post-test protocol was used - both the control and test groups first answered the same pretest, then were exposed to differing treatments for practice learning, and finally came back together to answer the same post-test. In order to eliminate practice effect, the average score per problem was used instead of the total score.
In spring 2001, the differential treatments were using the tutor versus using a printed workbook that included answers to problems as an appendix. The pre-post improvement in the score per attempted problem was 20.1% for workbook users (N=31) and 29.6% for tutor users (N=33), both the improvements being statistically significant (t-test 2-tailed p < 0.05).
In fall 2001, the differential treatments were using the tutor that provided both graphic visualization and text explanation versus using the tutor that provided only graphic visualization and no text explanation. The pre-post improvement in the score per attempted problem was 51.2% for the group that also received text explanation (N=33) versus 39.8% for the group that only received graphic visualization (N=33), both improvements being statistically significant (p < 0.05).
In fall 2002, the differential treatments were using the tutor that provided graphic visualization (but no text explanation) versus using one that did not provide any explanation except for the correct final answer. The pre-post improvement in the score per attempted problem was 53.1% for the group that received graphic visualization (N=24) versus 33.4% for the group that received no explanation (N=24), both improvements being statistically significant (p < 0.05).
Findings: Students who used the online tutor learned better than those who used a printed workbook. Clearly, online tutors are at least as effective as printed workbooks, the traditional source of exercise problems.
Students who received graphic visualization and text explanation learned better than those who received only graphic visualization. This confirms earlier result in literature that in order to be effective, visualization must be extended with explanation.
Students who received only graphic visualization still learned better than those who received no explanation. This highlights the importance of providing explanation in online tutors.
Recommendations for engineering education: Online problem-solving tutors are good substitutes for the traditional problem-solving workbooks, since they can provide instant feedback. The effectiveness of such tutors can be improved by supplementing graphic visualization with text explanation.
Acknowledgments: Partial support for this work was provided by the National Science Foundation under grants DUE-0088864 and CNS-0426021.
Author 1: Amruth N. Kumar amruth@ramapo.edu Article Link: http://portal.acm.org/citation.cfm?id=1047422&coll=portal&dl=ACM&CFID=9122936&CFTOKEN=32438184