Dive Brief:

• ChatGPT, an AI-driven natural language processing tool created by Microsoft-backed OpenAI, failed an accounting exam and came up short to students who were tested on the same questions, according to recent research from Brigham Young University.
• Researchers at BYU, along with 186 other accredited universities, co-authored over 20,000 accounting exam questions to ask ChatGPT. The questions covered accounting information systems, auditing, financial accounting, managerial accounting and tax, and varied in difficulty and type — true and false, multiple choice, short answer, etc.
• Students scored an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. Additionally, the AI bot did worse on tax, financial, and managerial assessments, with researchers concluding this could have been because ChatGPT struggled with the mathematical processes required for the latter category, the research said.

Dive Insight:

Although overall human students performed better than the technology, on 11.3% of questions ChatGPT scored higher than the student average, doing particularly well on AIS and auditing.

In terms of question type, ChatGPT did better on true and false questions, getting 68.7% of them correct, and multiple choice questions, with a total of 59.5% of questions answered correctly. 

Meanwhile, the chatbot struggled with short answer questions, only answering between 28.7% and 39.1% of them correctly.

“In general, higher-order questions were harder for ChatGPT to answer. In fact, sometimes ChatGPT would provide authoritative written descriptions for incorrect answers, or answer the same question in different ways,” the research said. 

ChatGPT has been catapulted into the spotlight in recent months. The tool has the ability to interact in a conversational way, producing responses to questions across a number of subject areas and CFO Dive recently engaged  with the chatbot itself, asking if the popular tool was ready to take over the duties of finance chiefs.

“While AI tools like myself can perform many tasks related to financial analysis, forecasting, and reporting, I don’t think we have the potential to completely replace corporate CFOs anytime soon,” ChatGPT said. 

Because of its capabilities, the tool has also drawn concern regarding its potential to replace workers in their jobs. Some job loss or disruption is likely, but AI will also create new jobs as well, a process that has already started, according to recent reporting from Industry Dive sister publication CIO Dive.

With the chatbot fell short on open-ended and like questions in the BYU accounting experiment, it likely does not have the analytical skills necessary for an accountant today. 

“Lower level finance jobs, like identifying credit risks across a customer base or protecting against expense fraud, or embezzlement or money laundering, these are all automated tasks that something like ChatGPT can significantly impact and free up resources,” said Greg Selker, managing director and regional practice leader, technology of North America at Stanton Chase — a New York-based executive search firm — in an interview with CFO Dive. 

Selker, along with various other experts, asserts that business adoption of artificial technology tools such as ChatGPT isn’t likely to make CFOs unnecessary in the near or even distant future. 

“Could it be that at some point, we have artificial intelligence that would replace the decision making process? I don't know, now we're talking Star Trek kinds of stuff,” said Selker.