When Stan Calow was growing up, frugality was a way of life: "You spend as little as you need to, and then save everything else." So, the 58-year-old engineer and U.S. Army veteran from Kansas City, Mo. always hated spending money.

It took his financial planner, Cindy Richey, to drill the point into him that it was actually okay to enjoy his savings once in a while.

After much prodding, the message finally got through. Calow and his wife just returned from a trip to France, touring the chateaux of the Loire Valley, just like they had always dreamed.

Says Calow, who learned about the fragility of life by serving in Kosovo: "I wanted to live life while I'm still young enough to enjoy it."

It's a tricky dilemma for many of us. As much as pundits tell us to scrimp and save and sacrifice for the future, when is it actually okay to spend a little on yourself and enjoy this life that passes all too quickly?

Indeed, according to a new survey, many of us are not enjoying it enough.

When Wells Fargo asked affluent Americans about what they regretted most about their finances, 15 percent said "not having enjoyed their money more".

It is an honest answer that you do not often encounter in financial surveys. After all, splurging on yourself is typically seen as selfish and gauche.

But as some planners point out, it's your money and you should not be made to feel bad about enjoying it occasionally.

"People are so nervous about outliving their money, and sometimes they shoot too far in their saving," says Joe Nadreau, director of innovation and strategy for Wells Fargo Advisors.