The biggest pet peeve for most millionaire investors are advisors who don't call them back promptly, according to a study released Tuesday by the Spectrem Group.

Among the main reasons cited for leaving their financial advisor, 73% of respondents said the top reason was an advisor not returning phone calls in a timely manner. In that vein, other big reasons included unhappiness with slow email response (57%) and an advisor who wasn't proactive in contacting them (56%). Surprisingly, these communication-related gripes seemed to hold equal or greater weight to bail on an advisor as failure to provide good ideas and advice (57%). 

The Millionaire Investor 2010 study, which covered investors with a net worth between $1 million and $5 million, also found 72% of millionaires expect their advisor to call back within 12 hours. But that's not fast enough for some clients, as 26% expect a callback within two hours and 14% expect a response in one hour.

"Advisors who don't respond promptly to their millionaire clients' calls are playing with fire," said George H. Walper, Jr., Spectrem Group's president. "Indeed, slow callbacks rank as the number one reason millionaires switch advisors. Importantly, it's also a very simple one to address."

Only 37% of respondents were satisfied with advisors responding to email by the next day, and 16% expect a response within two hours and 8% look for a reply within one hour.

Nearly all respondents listed honesty and trustworthiness as their top reason for choosing a new advisor, with 96% looking for an advisor who keeps clients informed of what they are doing and 90% seeking an advisor with a strong investment track record.