As American investors rebuild portfolios depleted during the financial crash of 2008, some demographic groups are doing better-or worse-than others, according to a new report.

The report by Hearts and Wallet, a research firm specializing in retirement market trends, indicates that while overall U.S. investor wealth is on the rise, there has also been a shift in the distribution of that wealth among various demographic and economic market segments.

Total U.S. household investable assets were at $30.2 trillion at year-end 2010, with retirement assets at $10.7 trillion and taxable assets at $19.5 trillion, according to the report. At the same time, taxable assets have climbed steadily since 2004, from $12.9 trillion to $19.5 trillion. Meanwhile, retirement assets have been recovering from their 2008 stumble to $8.1 trillion and have surpassed a 2007 peak of $10.5 trillion.

Out of 120 million households, about 30 million have at least $100,000 in investable assets, according to the report. That leaves 90 million households that have not reached the milestone, up from 82 million before the 2008 market slide.

"The 2008 downturn hit households very unevenly, and that may prove true again with continued market turbulence," said Laura Varas, Hearts & Wallets principal. "In intuitive terms, about 1 in 8 of the households now 'in' the $100,000 to $250,000 segment moved down from a higher investable asset segment."

Another notable find in the study: Investors moving down in assets have changing views of financial advisors and their firms based on their fluctuating fortunes. "For some households, the drop in asset segment was because of personal investment choices, and a segment of those are now seeking the help of a professional financial advisor, a behavioral segment identified by Hearts & Wallets as "upshifters,'" Varas said.

Other households suffered the decline while having professional financial advice, Varas added. "Some 'downshifters' now question the value of that advice and whether they might make better choices themselves," she said. "We were already seeing investors want to better understand the value of what they are paying for. This disruption is accelerating that trend."

Many affluent asset segments shrank in numbers,  according to the study:

The $2 million to $5 million and the $250,000 to $500,000 segments were particularly hard hit, according to the study. The $250,000 to $500,000 asset segment now totals 6.4 million households, down about 25 percent from 8 million in 2009. This segment is concentrated in the 45 to 54 and 55 to 64 age groups. This group controls $2.5 trillion in assets, down from $2.9 trillion, with the decline occurring in retirement accounts and managed investments.

The $1 million to $2 million and $100,000 to $250,000 segments grew, as they received households that shifted down into these asset segments.

Among the poorest Americans, the median household with very little to lose lost nothing. The median household in the 25th to 49th wealth percentiles lost $10,700. The 50th to 75th percentiles (corresponding to the $50,000 to $100,000 wealth segment), lost $40,000. For the 75th to 89th percentiles (from above $100,000 to just under $500,000), the median household lost $134,000. The worst one-quarter of households lost $261,000, but the best one-quarter actually improved relative to peers.

"The shifting of assets has been dramatic, illustrating the challenge in offering asset-based pricing services to investors," said Chris Brown, a Hearts & Wallets principal. "How much money someone has at a point in time is not a particularly insightful descriptor of their interests or concerns. The financial services industry needs to understand this when developing investor servicing models."

-Jim McConville