With about a third of retirees dealing with some level of cognitive impairment or dementia, finding the right time to transfer control of finances to a trusted agent is paramount to preventing costly financial mistakes or exploitation, according to a new report.

But while a survey of investors 55 and older found that most them could identify a trusted agent, the majority said they worried that they may delay transferring control and that the delay may result in financial losses.

The cognitive decline survey polled 2,489 participants in the Vanguard Research Initiative, which is a voluntary panel of older account holders at Vanguard Group, according to the brief “Transferring Control of Finances: Timing Poses a Risk” by John Ameriks, head of the firm’s quantitative equity group, and published by the Center for Retirement Research at Boston College.

Seventy percent of respondents said their agent would be a child or child-in-law (they were not permitted to name a partner or spouse), 10% chose a sibling and 9% a trustee or institution, the survey said. The vast majority said their agent would be either very good or excellent at understanding their needs and desire, understanding their financial situation and pursuing their interests. And 76% felt their agent would be available when the time comes, the survey said.

However, that’s as far as retirees’ confidence went.

“For an agent to capably protect a person’s finances from the effects of cognitive decline, the transfer of financial control should occur before people make irreversible mistakes,” Ameriks wrote in the survey brief. “Many factors make it challenging to transfer control at the right time, including the elusiveness of cognitive decline.”

When given a hypothetical scenario of mild cognitive decline, respondents were charged with deciding when to transfer control to their trusted agent. Only 8% wanted to transfer control immediately even though that’s the option that would reduce the risk of financial mistakes the most, and on the flip side only 8% wanted to wait until they’d completely lost their ability to manage their finances.

The majority—84%—wanted to hit a sweet spot where they were willing to decline a little further before the transfer, but didn’t want to wait until their impairment grew to the point where they were more at risk of losses.

“Strikingly, respondents think the chances of missing the optimal timing are significant,” Ameriks wrote. “When asked about the subjective probability of having the transfer at the wrong time, the average respondent thinks there is a 35% chance of the transfer occurring too late and a 24% chance of it occurring too early.”

Reasons for a delayed transfer included the retiree not noticing their own decline, or getting to that point but not wanting to give up control. Other reasons were the agent not noticing the decline or not being available when the time came.

And the damage of a delayed transfer was estimated by the retirees to be quite high—18% of their total wealth.

The primary reason given for a premature transfer was the retiree’s concern that the agent would take control against their wishes, and the participants anticipated this would cost 10% of their total wealth.

“The survey respondents’ desire to keep control while still capable exposes them to the risk of a delayed transfer,” Ameriks wrote. “These results suggest that any measures that can help secure the optimal timing of the transfer of control – e.g., regular monitoring of cognitive abilities – can go a long way to protecting older Americans’ financial well-being.”