The AI chatbot, ChatGPT, developed by OpenAI, has the ability to detect and correct bugs in computer code, and it performs even better when engaged in conversation. Researchers from Johannes Gutenberg University in Mainz, Germany, wanted to test ChatGPT’s ability to fix errors in code and compare it with other AI-powered coding support tools. To do so, they asked ChatGPT a series of small but challenging programming questions taken from the QuixBug benchmark dataset, all written in the Python programming language.
ChatGPT correctly answered 19 out of 40 queries, which is comparable to other deep learning-based code fixing approaches. However, the researchers also tested ChatGPT‘s conversational interface, asking follow-up questions that a user would ask when inserting corrected text into a programming tool.
During the experiment, the researchers discovered that ChatGPT‘s conversational interface improved its ability to fix errors in code. In fact, when the researchers utilized ChatGPT‘s conversational interface to ask follow-up questions, ChatGPT correctly solved the issue in 31 out of the 40 queries. This was an unexpected result, and the researchers believe that ChatGPT or similar systems could be useful as an additional troubleshooting tool for programmers in the future. However, Alan Woodward at the University of Surrey cautions that human oversight is still necessary since AI is not infallible.
Woodward also believes that AI-powered coding support tools could be used not only to check static code but also to provide suggestions for improving code in software development programs.
Reference: Arxiv, DOI: 10.48550/arXiv.2301.08653
