A former Cold War code breaker may have cracked the tax code for hedge funds.

James H. Simons, who became a billionaire when he turned his extraordinary mathematical ability from defense work to investing, has deployed an unusual strategy at Renaissance Technologies LLC to skirt hundreds of millions of dollars in taxes for himself and other investors, said people with knowledge of the matter.

The Internal Revenue Service is challenging the technique, which it called “particularly aggressive,” without identifying the hedge fund in the dispute. It is demanding more tax payments from investors in Renaissance’s $10 billion Medallion fund, the people said.

Renaissance sought to convert profit from Medallion’s rapid trading into long-term capital gains, said the people, who spoke on condition of anonymity because the dispute hasn’t been made public. The top federal rate on long-term gains is about half that on short-term.

Versions of the strategy, which involved shifting ownership of Medallion’s portfolio to banks including Barclays Plc, were used for most of the past decade, one person with knowledge of the matter said.

The case highlights how hedge-fund and private-equity managers use loopholes to exploit the government’s preferential treatment for long-term investing income. If East Setauket, New York-based Renaissance prevails in its legal dispute with the IRS, dozens of other funds would probably take steps to mimic the firm’s strategy, according to tax advisers.

‘Aggressive Strategies’

“If they win, that will signal to the rest of the hedge-fund community that aggressive strategies can work,” said Steven Rosenthal, a fellow at the Urban Institute in Washington and a former tax lawyer at Ropes & Gray LLP.

Jonathan Gasthalter, a spokesman for Renaissance, declined to comment, saying “the dispute is ongoing and being handled in the appropriate forum.”

Kerrie Cohen, a spokeswoman for Barclays, declined to comment. The IRS declined to comment, citing confidentiality laws.

The disagreement between the IRS and Renaissance traces back at least to 2010, when the IRS publicly criticized an unnamed hedge fund for using the strategy. It was referring to Medallion, the people said.

Simons, 75, retired from Renaissance that year as one of the best-performing managers of all time and now serves as non-executive chairman.

Testing ‘Lucifer’

While working for the nonprofit Institute for Defense Analyses at its research center in Princeton, New Jersey, in the 1960s, Simons was a code breaker for the National Security Agency. In 1973, he helped International Business Machines Corp. test a cipher, named “Lucifer,” that its researchers had developed.

A philanthropist and former math professor who has championed better math and science education, Simons has a fortune estimated by the Bloomberg Billionaires Index at $12 billion. He didn’t respond to a written request for comment.

The IRS contends that Medallion’s arrangement with the banks, in which the fund owned option contracts rather than the underlying financial instruments, is a ruse and that the fund investors owe taxes at the higher rate. Renaissance takes the position that the trades were not tax-motivated, were consistent with current law and were done for legitimate business reasons, according to a person with knowledge of the matter.

Private Dispute

The dispute hasn’t spilled over into a public forum such as U.S. Tax Court. Unless they prompt litigation, taxpayer conflicts with the IRS are handled in secret and often take years to resolve. It couldn’t be determined whether Renaissance is still using the strategy, and how much the IRS wants investors to pay in back taxes.

A former Renaissance employee, who spoke on condition of anonymity, said it notified him years after he left that the IRS was challenging the tax technique, and he might have to pay more than $90,000 in additional taxes if the firm loses.

Renaissance assured him that the trade was a common and legitimate technique and that the IRS was trying to rewrite the rules retroactively, the former employee said.

A proposal this year from U.S. Representative Dave Camp, a Michigan Republican and chairman of the House Ways and Means Committee, would eliminate any possibility that the Renaissance technique could generate tax savings, Rosenthal said. That proposal, a plank in Camp’s effort to overhaul the tax code, would prevent owners of derivatives such as options from earning any long-term capital gains.

Frequent Trading

The top federal rate on long-term capital gains, derived from selling investments held for more than a year, is currently 20 percent, compared with 39.6 percent imposed on wages and investments of less than a year. Before this year, the short- term rate was 35 percent and the long-term was 15 percent. The Medallion fund trades stocks and futures so frequently that, absent the tax maneuver, it would generate mostly short-term gains, said the people with knowledge of the matter.

Although tax experts said some of the specifics of Renaissance’s strategy are unusual, figuring out ways to convert a hedge fund’s trading profits into income taxed at the lower, long-term gains rate is one of the holy grails for tax planners.

“It’s been going on since there’s been hedge funds,” said David Weisbach, a tax professor at University of Chicago Law School.

Offshore Investment

One increasingly popular technique is to set up an offshore insurance company that in turn makes the hedge fund investment, as top executives at billionaire John Paulson’s firm did last year. The move was the subject of a Bloomberg News report in February. At the time, Paulson declined to say whether it was done to avoid taxes.

Renaissance’s strategy involved buying an instrument called a “basket option contract,” from banks including Barclays, the people said.

IRS lawyers released an 11-page memorandum in 2010 describing the technique and outlining what they called a “particularly aggressive” example, without naming Medallion and Barclays.

The purpose of the memo was for top IRS attorneys in Washington to notify staff in the field about a newly discovered tax-avoidance technique. The IRS had been tipped off to the practice in 2008 by examiners from the U.S. Securities and Exchange Commission.

Stock Portfolio

As described in the memo and by people with knowledge of the matter, the transaction worked as follows: Barclays bought a portfolio of stocks and other instruments that fund managers at Renaissance wanted to trade. The bank hired the fund managers to oversee the portfolio, paying them a nominal fee.

Then Medallion bought an option with a term of two years, whose value was linked to the worth of the portfolio. Renaissance had full discretion to trade the securities in the portfolio.

Medallion could claim it owned just one asset -- the option -- which it held for more than a year, allowing any gain to be treated as “long-term” when its investors reported the income on their personal tax returns.

“The profits are just being transmuted, through the alchemy of derivatives, to a preferenced return,” said Urban Institute’s Rosenthal.

IRS Memo

Although the memo describes a specific basket option contract provided by Barclays, Renaissance entered into similar contracts with Deutsche Bank AG, said the people with knowledge of the matter. After the IRS published the 2010 memo criticizing the technique, Deutsche Bank stopped offering Renaissance versions of the option trades that had tax-reducing benefits, two of the people said.

Deutsche Bank spokeswoman Renee Calabro declined to comment.

Some of the people with knowledge of the option trades said they had at least one purpose unrelated to tax savings: they allowed Medallion to borrow more money from its banks than it otherwise could.

Robert J. Frey, who worked at Renaissance from 1992 to 2004, said he has heard about the dispute from current employees. They told him the firm hasn’t done anything wrong.

“These are people that were typically very conservative about that kind of thing,” he said in an interview. “There was never any desire to do anything that was operating on the edge. It’s a pretty honorable group of people.”

University Ties

Frey is now a professor at Stony Brook University, which has close ties with employees at Renaissance and is located near the firm’s headquarters on the north shore of New York’s Long Island.

Simons, who holds a bachelor’s degree from the Massachusetts Institute of Technology and a doctorate from the University of California, Berkeley, is a former chairman of the Stony Brook math department and has a geometric theorem named after him. He left academia in 1977 to manage money. At Renaissance, he assembled a team of Ph.D.’s who turned their math skills to buying and selling stocks and futures, creating computer programs to identify profitable trades.

Its flagship Medallion fund has returned about 80 percent annually before fees since 1988, according to the Bloomberg Billionaires Index.

Simons and other Renaissance employees own almost all of the fund. Medallion stopped new investments from outsiders in 1993 and kicked out most non-employee investors in 2005.

The Billionaires Index estimates that Simons has earned at least $10 billion from Renaissance over the years. Institutional Investor’s Alpha estimated that he earned $1.1 billion last year alone.

Ex-Treasury Official

Renaissance’s legal team in the dispute includes Kenneth W. Gideon, a former IRS chief counsel and former assistant Treasury secretary for tax policy. He’s now a partner at Skadden Arps Slate Meagher & Flom LLP in Washington. Gideon declined to comment, other than to confirm he is counsel to Renaissance in an IRS dispute.

Edmund S. Cohen, a lawyer at Winston & Strawn LLP in Washington, helped Renaissance set up the trades and supplied a legal opinion stating that the arrangements complied with tax laws, the people with knowledge of the matter said. Cohen declined to comment.

Tax planners started using derivatives to convert hedge funds’ short-term gains to long-term gains in the 1990s, said Alex Raskolnikov, a tax professor at Columbia University Law School. Congress tried to close the loophole in 1999, enacting a law allowing the IRS to disregard the tax effect of some derivatives, such as swaps and forwards, if they were economically akin to owning the fund directly.

‘End Run’

As described in the 2010 IRS memo, the basket option contract used by Renaissance represents an “end run” around the 1999 law, said Robert N. Gordon, president of Twenty-First Securities Corp., which advises clients on the tax implications of investments. Although the transaction is technically an option, it’s structured in such a way that it is economically almost identical to owning the portfolio, he said.

“We would love to have something to get long-term gains out of a hedge fund,” Gordon said, adding that the option described in the memo probably runs afoul of the 1999 law. “This thing doesn’t work.”

Autism Research

Since his retirement, Simons has focused on philanthropy. He is a major funder of autism research and is the founder of Math for America, which seeks to improve math education in public schools.

Simons has homes in Manhattan and on Long Island’s north shore, and owns a 222-foot yacht, Archimedes, named for the ancient Greek mathematician.

Simons and his wife, Marilyn, gave $9.6 million to Democratic groups in 2012, including a “super-PAC” that backed President Obama, according to data compiled by the Center for Responsive Politics.

During the same election cycle, Robert L. Mercer, the firm’s co-chief executive officer, gave $5.4 million to mostly Republican groups, the data show.

Renaissance has spent more than $2 million since 2002 lobbying Congress and the Treasury Department on taxes, according to lobbyist disclosure forms. Some of the lobbying has been for unspecified “tax issues affecting hedge funds.” None of the disclosure forms mention the IRS dispute.

Some of Renaissance’s lobbying has focused on proposed reforms that would restrict fund managers’ ability to claim a lower tax rate on some of the “carried interest” they earn from their jobs. The issue drew attention during last year’s presidential election when the Republican nominee, Mitt Romney, disclosed he had paid just 13.9 percent of his 2010 income in taxes. Romney benefited from a rule that allows him to count some of his compensation from running the private-equity firm Bain Capital LLC as long-term capital gains even when the “carried interest” doesn’t derive from his own investments.

Testifying before Congress in 2008, Simons was asked by U.S. Representative Elijah Cummings, a Maryland Democrat, whether he would “support repealing this tax loophole” and paying the ordinary rate.

“That would be OK with me,” Simons said.