A former Cold War code breaker may have cracked the tax code for hedge funds.

James H. Simons, who became a billionaire when he turned his extraordinary mathematical ability from defense work to investing, has deployed an unusual strategy at Renaissance Technologies LLC to skirt hundreds of millions of dollars in taxes for himself and other investors, said people with knowledge of the matter.

The Internal Revenue Service is challenging the technique, which it called “particularly aggressive,” without identifying the hedge fund in the dispute. It is demanding more tax payments from investors in Renaissance’s $10 billion Medallion fund, the people said.

Renaissance sought to convert profit from Medallion’s rapid trading into long-term capital gains, said the people, who spoke on condition of anonymity because the dispute hasn’t been made public. The top federal rate on long-term gains is about half that on short-term.

Versions of the strategy, which involved shifting ownership of Medallion’s portfolio to banks including Barclays Plc, were used for most of the past decade, one person with knowledge of the matter said.

The case highlights how hedge-fund and private-equity managers use loopholes to exploit the government’s preferential treatment for long-term investing income. If East Setauket, New York-based Renaissance prevails in its legal dispute with the IRS, dozens of other funds would probably take steps to mimic the firm’s strategy, according to tax advisers.

‘Aggressive Strategies’

“If they win, that will signal to the rest of the hedge- fund community that aggressive strategies can work,” said Steven Rosenthal, a fellow at the Urban Institute in Washington and a former tax lawyer at Ropes & Gray LLP.

Jonathan Gasthalter, a spokesman for Renaissance, declined to comment, saying “the dispute is ongoing and being handled in the appropriate forum.”

Kerrie Cohen, a spokeswoman for Barclays, declined to comment. The IRS declined to comment, citing confidentiality laws.