The way retirement is supposed to go is that you scrimp and save for decades while you are working, and then you get to enjoy years of luxurious, worry-free leisure.

But here is a little tip: Fun does not just happen. You have to budget for it.

According to new data from Merrill Lynch and Age Wave, 77 percent of retirees have hardly planned at all for their first five years of leisure activities. Two-thirds of those surveyed for the "Leisure in Retirement: Beyond the Bucket List" report have not budgeted for any travel. And 45 percent have not even guessed at how much leisure activities might cost throughout their golden years.

This is not just a minor bookkeeping oversight. If you don't factor in the fun, it could blow a meteor-sized crater in any financial plan.

"These costs are significant, and you absolutely have to factor them in," says Cyndi Hutchins, director of financial gerontology for Bank of America Merrill Lynch. "People just can't seem to wrap their heads around what their future lives are going to look like."

Financial planner Douglas Kobak, of Conshohocken, Pennsylvania, recently met a client with grand plans of retiring in two years, buying a winter home in Florida and traveling the world. But when Kobak sat down with him and actually calculated the costs of the dream trip, the client had to rethink the whole plan -- as well as his projected retirement date.

"It wasn't what he wanted to hear," Kobak said.

Another new survey confirms that we have forgotten about fun in our golden years. If given another 30 years of life, 56 percent of respondents vowed to "travel extensively," and 35 percent dreamed of "living in a different place," according to the study by insurer Allianz Life, in conjunction with the Stanford Center on Longevity.

But because our lives have taken different paths, almost a third of us are filled with regret about how things turned out. Talk about depressing.

Here are a few pointers from financial planners about how to budget for fun -- and avoid any nasty accounting surprises before it is too late.