The prolonged economic downturn has become the top worry for most wealthy investors, according to new research report released Tuesday by Chicago-based financial research firm The Spectrem Group.

An estimated 80 percent of millionaire investors surveyed cited the sluggish first-quarter economy as a concern compared with 70 percent in the same period in 2011. Similarly, an estimated 81 percent of mass affluent investors with net worths between $100,000 and $1 million were concerned about this year's first-quarter economic performance compared with 74 percent last year.

Only ultra high-net-worth investors -- those with at least $5 million in assets -- didn't list a slow economic recovery as their chief concern. Instead, an estimated 82 percent of members in that group cited the national debt. Second on their list of concerns was a contentious political environment, according to 80 percent. A weak economy came in third, with 77 percent, compared with 69 percent for the same period last year.

"With the unemployment rate remaining high, the continuing volatility in the stock market and gasoline prices, and a high-stakes presidential election looming, wealthy investors are anxious about the country's outlook," says George H. Walper Jr., president of Spectrem Group. "You can expect a strong turnout among these voters."

Anxieties among the affluent about maintaining their own financial position and having enough money for retirement also increased from a year ago. In addition, a strong majority worry about tax increases, including70 percent of ultra high net worth investors,65 percent of millionaire investors (net worths of $1 million to $5 million), and64 percent of mass affluent investor.

But tax concerns are not translating into major investment changes. Just 40 percent of the wealthiest, 28 percent of millionaires and 25 percent of mass affluent investors are making investment changes because of possible tax increases. "Overall, wealthy investors are more confident about their own finances, with roughly half optimistic that their personal financial situation will improve in 2012," Walper said.

Spectrem Group in the first quarter this year surveyed 1,319 mass affluent, 1,303 millionaire and 482 ultra-high-net worth investors, with a focus on investment attitudes and behaviors.