Investors' confidence in their advisors is declining in many respects, despite the fact that the market is starting to come back, according to a new study by Spectrem Group.

Spectrem Group measured investor satisfaction from several perspectives for ultra-high-net-worth investors: those with $5 million to $25 million in net worth not counting their primary residence; for millionaire investors with between $1 million and $5 million, and for the mass affluent, those with $100,000 to $1 million in assets.

For UHNW investors, 73 percent are satisfied with their financial advisors, a significant decrease from the 80 percent who were satisfied last year and 81 percent in 2010.

Among millionaire investors and the mass affluent, satisfaction levels improved somewhat. For millionaire investors, 72 percent said they are satisfied now compared to 70 percent the last two years, and for the mass affluent, 69 percent say they are satisfied this year compared to 66 percent in 2011 and 61 percent in 2009.

In what Spectrem calls a lukewarm vote of confidence, just half of the mass affluent investors say they would follow their advisor to a new firm, down from 59 percent in 2011. Likewise, the percent was down for the other two groups. Fifty-five percent of millionaire investors would follow their advisor, down from 61 percent last year, and among UHNW investors, 52 percent would follow their advisor, down from 60 percent last year.

"Advisors clearly are not moving the dial in the right direction when it comes to satisfying enough of their clients," says George H. Walper Jr., Spectrem Group president. "Perhaps even more telling than satisfaction levels, more than one quarter of affluent investors in all three wealth segments think they do a better job of investing than their advisors."

"These findings should be a wake up call that advisors need to step up their game to attract and retain such important clients," he adds.

When asked about specific products and services, Spectrem says investors are not overjoyed with their advisors' performances. When asked about face-to-face meetings, a rating of excellent was given by 51 percent to 54 percent in the three categories of wealth.

In the three categories, 42 percent to 45 percent rate their advisors as excellent on the financial plans they prepare. Only 28 percent to 22 percent rate their advisors excellent on the preparation of newsletters.

"To reverse this trend and develop loyalty, advisors need to demonstrate sophisticated, in-depth knowledge about taxes, financial planning and related issues, while working more collaboratively with investors who may well have identified investing opportunities on their own," Walper says.

-Karen DeMasters