Just 31% of pre-retirees and retirees could be considered financially literate when it comes to retirement income, according to a recent study by the American College of Financial Services.
But maybe Americans shouldn’t feel too bad about it. When two researchers at the college, Chet Bennetts and Eric Ludwig, posed the same 36 study questions to ChatGPT, the large language model chatbot did only marginally better.
“ChatGPT has passed the bar exam. ChatGPT passed the CFA and CPA exams. We thought it would score something like 90%, or at least do very well because it’s this all-knowing database of information and has passed these other difficult exams,” said Bennetts, who is also the college’s program director for the CFP and Chartered Financial Consultant designations.
ChatGPT scored just 45%. Better than the average American, but not by much.
“We were stunned,” he continued. “Could it be that we were asking questions that were so abstract that even this all-knowing entity, AI, couldn’t answer them correctly? Or is it that because there’s so many variations of the right answer for people when they’re making these decisions that there’s a lot of subjectivity that really is only moderated by a human?”
Intrigued, Bennetts and Ludwig conducted another trial. This time, they trained ChatGPT using the textbooks that are used in the three courses needed for a Retirement Income Certified Professional (RICP) designation.
They then opened a new session so the chatbot couldn’t reference its previous answer and ran the quiz again. This time, the results were much more acceptable: ChatGPT was wrong only once, and scored 92%.
“When we used the training materials that we train advisors on to help guide client decisions, and it had access to that knowledge, it did great,” said Ludwig, who is also the program director for the RICP as well as the director for the college’s Center for Retirement Income. “Which means the financial advisor is very important because the advisor is able to ask the questions and clarify the answers in a way AI alone can’t get correct.”
The online survey of nearly 4,000 adults, aged 50 to 75, has been conducted every three years since 2014. Although the 2023 study examined different subsets of its respondent pool, the percentage of participants passing the test never rose above 50%, and that was for those with more than $1.5 million in assets.
Other high achievers were college grads, 40% of whom passed. Also retirees aged 70 to 75, 38% of whom passed, although only 33% of all retirees passed.
“This was pretty much in line with previous surveys,” Ludwig said, adding that he hoped that the SECURE Acts 1.0 and 2.0, which should have improved retirement outcomes for many Americans, would also be simple enough to be readily understood by average citizens. “So we were looking at whether people are getting better at this or not. And it turns out, they’re not.”
Having an advisor did not change the literacy scores either, the researchers found, as 36% of unadvised participants with more than $100,000 in assets passed the quiz, a percentage that hiccuped up to 37% for the same demographic with an advisor.
But the presence of an advisor for people with less than $100,000 in assets had a more significant impact. Without an advisor, only 25% of participants passed. With an advisor, 37% did.
“So the role of the advisor relationship in these complex areas of retirement knowledge did have an impact, and it was significant in that it doesn’t matter that a financial advisor is present when it came to the retirement income literacy score,” Bennetts said. “There were other factors around having an advisor that played into less anxiety and stress.”
For example, participants with an advisor experienced 20% less anxiety and 25% less stress—and scored slightly better on the survey.
“All of those components just really hearken to how important the advisor is and how we are probably not going to be replaced by the machine anytime soon—because the machine does not have the emotional response that the advisor brings and is able to benefit our respondents,” he said.
In addition, Ludwig pointed out, during the market volatility of 2022, respondents with an advisor were two times more likely to maintain their portfolio as those without.
“To see that they’re helping people maintain their portfolios during these times of market volatility, I think that’s another thing where advisors can hold their heads up and feel really good about it.”