Earlier this month, New York City public schools officially opted to ban ChatGPT from school devices and WiFi networks over concerns of cheating. Putting OpenAI‘s revolutionary chatbot to the test, law professors at the University of Minnesota conducted a study in which they blindly graded graduate-level tests taken by ChatGPT alongside that of actual students.
The exams given to ChatGPT included a total of 95 multiple-choice questions and 12 essay questions. In general, the bot was able to perform better on the essays, though scores were inconsistent. ChatGPT was able to understand the basic concepts but struggled to apply the knowledge in an applicable and nuanced way like a real law student would. The end result was an average grade of C+ for the AI program. While it would be enough to earn a law degree, it still falls below the human average at the university of B+.
Both camps warned that schools should limit the use of technology to prevent ChatGPT-based cheating. They also recommended altering the questions to either discourage AI use (such as focusing on analysis rather than reciting rules) or increase the challenge for those people leaning on AI. Students still need to learn "fundamental skills" rather than leaning on a bot for help, the University of Minnesota said. The study groups still believed that ChatGPT could have a place in the classroom. Professors could teach pupils how to rely on AI in the workplace, or even use it to write and grade exams. The tech could ultimately save time that could be spent on the students, Terwiesch explains, such as more student meetings and new course material.
“Alone, ChatGPT would be pretty mediocre law student,” said study collaborator Jonathan Choi said in a statement. “The bigger potential for the profession here is that a lawyer could use ChatGPT to produce a rough first draft and just make their practice that much more effective.”
