"We take these calls, millions of calls every year," said Jeanne Thompson, a Fidelity vice president. "We see they have taken loans, and they don't have enough to retire."
A direr problem is with those who have an outstanding balance when they lose or change jobs. They must repay their loans immediately or face tax penalties on top of credit problems.
"The vast majority of money is actually repaid, on the order of 85 percent of it," says Harvard's Madrian. "But for a smaller subset of people, it can be a problem."
Legislation to change 401(k) loan provisions is unlikely at this point, Madrian said.
"It would be easier if you had some companies get rid of the option and show the employees were better off," she said. "Absent some more compelling data, it's going to be hard to shift the policy landscape on that front."