Jan Pearce, Chris McDonald, Durell Bouchard, Amey Karkare, Dennis Bouvier, Paul Denny, Brett Becker, James Prather, Brian Harrington, Ray Pettit, Amir Kamil, and Peter-Michael Osera.
Working group report:
Paper available here:
https://brettbecker.com/publications
Corpus for download:
Bibfile for corpus on Programming Error Messages (1965 - present):
complete-corpus.bib
Contribute!
Do you have a publication or resource on Programming Error Messages that we don't have?
Email Brett here http://brettbecker.com/contact.
Abstract:
Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen an increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance, track student progress, and tailor learning plans. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form.
In order to help the computing education community (and related communities) to further advance work on programming error messages, we present a comprehensive, historical and state-of-the-art report on research in the area. In addition, we synthesise and present the existing evidence for these messages including the difficulties they present and their effectiveness. We finally present a set of guidelines, curated from the literature, classified on the type of evidence supporting each one (historical, anecdotal, and empirical). This work can serve as a starting point for those who wish to conduct research on compiler error messages, runtime errors, and warnings. We also make the bibtex file of our 300+ reference corpus publicly available. Collectively this report and the bibliography will be useful to those who wish to design better messages or those that aim to measure their effectiveness, more effectively.
Citation:
@inproceedings{becker2019compiler,
author = {Becker, Brett A. and Denny, Paul and Pettit, Raymond and Bouchard, Durell and Bouvier, Dennis J. and Harrington, Brian and Kamil, Amir and Karkare, Amey and McDonald, Chris and Osera, Peter-Michael and Pearce, Janice L. and Prather, James},
title = {Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research },
booktitle = {2019 ITiCSE Working Group Reports},
series = {ITiCSE-WGR '19},
year = {2019},
isbn = {978-1-4503-6895-7/19/07},
location = {Aberdeen, Scotland UK},
pages = {},
numpages = {33},
url = {http://doi.acm.org/10.1145/3344429.3372508},
doi = {10.1145/3344429.3372508},
acmid = {},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {compiler error messages; considered harmful; CS1; CS-1; design guidelines; diagnostic error messages; error messages; human computer interaction; HCI; introduction to programming; novice programmers; programming errors; programming error messages; review; run-time errors; survey; syntax errors; warnings},
}