A minority of pre-retirees plan to retire to their dream home, according to a Voya Financial survey on housing and retirement.

Only 12 percent of pre-retirees plan to save to buy their dream home. Fifty-seven percent say they plan to remain consistent with their current housing budget and 27 percent plan to scale down rather than up, the survey of 1,000 Americans shows.

At the same time, most homeowners with mortgages plan to pay off their mortgages before retirement but slightly less actually manage to do so. Eighty percent of pre-retiree homeowners with a mortgage are optimistic they will pay off their mortgage before retirement, but 26 percent of retirees still have mortgages, says Voya.

While housing costs are a concern for 15 percent of those surveyed, health-care costs outweigh them by causing concern for 41 percent of people. But in contrast, if consumers received an unexpected $300,000 for their retirement, 35 percent would use it to pay off a mortgage and 23 percent would save the money for future health-care costs.

The good news from the survey is that 63 percent of the couples agree where they want to retire. Thirty-seven percent of the couples do not agree or have not talked about it. Surprisingly, 23 percent of those already retired are not in agreement about where they should be or still have not discussed it.

Relationships remain the key factor in deciding where individuals want to live in retirement. Thirty-eight percent of respondents say proximity to family and friends is the most important influence, compared to 12 percent who say cost of living is the determining factor. The remainder listed a number of factors.

“Retirement goals are personal, and each individual or couple is on their own journey. The one constant, however, is to make sure you map out a plan to reach your destination,” says Rich Linton, president of Large Corporate and Retail Wealth Management Markets at Voya Financial.