Forget all the arguments about whether or not a wealth tax on the ultra-affluent is constitutional. There's a far more basic, logistical reason why a wealth tax will never materialize in America.

The IRS simply lacks the manpower and infrastructure to appraise all the assets of a few thousand wealthy individuals and families on an annual basis, according to Andy Friedman, proprietor of The Washington Update and a widely respected analyst of national policy.

There is "no chance of a wealth tax," Friedman told attendees at the Investment & Wealth Institute's annual Investment Advisor conference in New York City on February 12. "You only have your [entire] wealth appraised once in your life."

That occurs when you die. Any advisor who has helped a client's family settle a $4 million to $7 million estate knows that the process is arduous, expensive and time-consuming. It can often take several years.

The financial lives of ultra-wealthy Americans with net worths exceeding $50 million are far more complex than those of most upper-middle clients. The IRS "couldn't handle it," said Friedman, who ran the tax practice at the giant Washington law firm Covington & Burling.

The idea of appraising one's wealth, including real estate, private companies, collectibles and other non-transferrable assets typically owned by the wealthiest Americans, every year is unrealistic, in Friedman's view. Such an undertaking would entail expanding the appraisal and accounting professions substantially in both the private sectorand at the IRS.

Asked about all the talk of 70 percent income tax rates on the super-rich, Friedman said it was highly unlikely. Taxes may head higher but it "won't be as draconian."

Were the Democrats to gain control of the White House and both houses of Congress in 2020, he considered it quite likely that they would seek to restore the 39.6 percent top marginal rate that existed during the Clinton and Obama administrations. It is possible that people earning more than $5 million a year could see a 50 percent tax rate, Friedman said, but he doubted such a tax would hit people earning in the low seven figures.

For all the talk about higher taxes, it remains to be seen how Americans react when they file their taxes later. Millions in the upper-middle class are expected to pay higher taxes this year and that could influence how they view the issue going forward.