At a 2007 fundraiser for Sen. Hillary Clinton's presidential campaign, Warren Buffett took aim at a U.S. tax system that would allow him to pay a lower tax rate than that of his secretary and house cleaner.

Noting that he was taxed at a capital gains rate of 17.7% on the $46 million he made the previous year, while his secretary was taxed at a rate of 30% on $60,000, he told the crowd, "If you're in the luckiest 1% of humanity, you owe it to the rest of humanity to think about the other 99%."

Many in that 1% grouping may beg to differ. While many wealthy individuals are charitable and philanthropic, there's no evidence that they're chomping at the bit to pay more in taxes. If they're going to make charitable donations, many say they'd rather do it themselves than give it to the government to dole out.

"We look at Washington and see enormous inefficiencies and are not willing to feed our hard-earned income into that inefficient system," says Robert Fragasso, president of Fragasso Financial Advisors in Pittsburgh, and a wealthy individual in his own right.

Not surprisingly, those who believe America's richest should pay more are those who earn less. A Quinnipiac University poll conducted a year ago found that 60% of Americans think the Obama administration should repay the gaping budget deficit by raising taxes on those who make $250,000 or more. Apparently, however, someone forgot to convey this wish to Congress. In December, a month after his party suffered historic losses in Congress, President Barack Obama signed a compromise tax bill that extended the controversial Bush-era tax cuts-the biggest beneficiaries of which were the wealthy. The package, brokered by Obama and Republican leaders, angered many progressive Democrats, but their efforts to scale back the bill's benefits for taxpayers at the highest income levels failed, and the measure passed with bipartisan support.

Despite the eleventh-hour win, many wealthy individuals still believe tax rates in the U.S. are too high, says Kathy Boyle, founder and president of Chapin Hill Advisors, a financial planning firm in New York.

"We hear a lot of anger, we hear a lot of frustration, especially in states like New Jersey," Boyle says. "Gov. Chris Christie has taken a lot of heat for things like cancelling the tunnel project, but the people who live in New Jersey who are high-net-worth are applauding him."

Boyle says she has baby boomer clients who have made a lot of money and have fixed costs of about $500,000 to $600,000 a year, and they resent that their tax bills are eating into their income.

"They've created this lifestyle, and they don't like that they'd have to cut back in their 60s because the government is trying to get a bigger piece of what they earned," she says. "A lot of them are just saying, 'I'm going to keep my income under $250,000.' I hear this from a ton of people."

These are people in the $10 million range, with multiple homes and Porsches, she says. They're concerned they haven't made back what they lost in 2008. She's watching them rein in their philanthropic dollars, sell their jets and consider getting into shared jet arrangements, even pulling their kids out of Montessori schools. One client, with a net worth of $100 million, let one of his golf memberships go. Another, who had $40 million to $50 million in net worth, made a large gift of $7 million to $8 million to his college alma mater, and he had to go back and ask them to agree to an installment plan. Boyle says she has two clients who are executives-turned-handymen, who used to do work for wealthy families in Westchester and Long Island, N.Y., and both have said their estate-type clients are no longer calling. Some have had to put off selling the house and move to a lower-tax state for retirement because their adult children have moved back home.