Economic turmoil, investment worries and inflation fears are making high-net-worth people less confident about their financial situations. It's a sharp reversal from one year ago when they had record-high levels of optimism, but according to the ninth annual Phoenix Wealth Survey, only 54% of respondents reported feeling wealthier in 2008 versus 81% in 2007. And only 25% said they were "very optimistic" about their financial future, a nine-percentage-point drop from last year.

   "The mood has shifted from optimism to concern, and even the wealthiest Americans are worried about losing ground, particularly when it comes to retirement," said Walt Zultowski, senior vice president of research and concept development at The Phoenix Companies, a Hartford-based life insurance and annuities company whose survey contacted more than 1,900 people with $1 million or more in net worth (excluding their primary residence).

   Forty percent of respondents said they're worried their assets will be depleted too quickly or that they won't be able to live comfortably on their retirement income. That number was 36% last year.

   Similarly, higher percentages of people this year are worried that inflation and poor investment performance will erode their asset base.

   More than half (58%) believe the U.S. economy is in a downturn and the worst is yet to come.

   Despite the angst, the survey found that most wealthy consumers (89%) who have a primary financial advisor report being satisfied with their advisor. More than half (56%) have been with the same advisor for six or more years, and just 8% say they plan to look for a new advisor in the next 12 months.

   Of those who plan to shop around for a new advisor, 42% cite less-than-expected investment returns as a reason, and 31% say fees are too expensive, a huge jump from 2007 when just 8% cited fees.

   "This suggests that in a down market, high-net-worth individuals assess their advisors from a cost/benefit point of view, along with other criteria," Zultowski said.