A federal judge in Texas ruled that former billionaire Sam Wyly defrauded the IRS by shuffling assets among a network of offshore trusts to evade millions of dollars in taxes.

U.S. Bankruptcy Judge Barbara Houser in Dallas said Tuesday that Wyly, a “sophisticated” businessman, was aware of the efforts of his advisers to hide his assets. Houser roundly rejected arguments that the 81-year-old entrepreneur was simply following orders from his own employees.

“The court does not believe that the law permits Sam to hide behind others and claim not to have known what was going on around him,” she said.

The ruling is the latest blow to Wyly, once a fixture in Texas high society, in a string of legal battles with U.S. agencies that threatens to wipe out the fortune he amassed over a lifetime.

The Internal Revenue Service claimed to be the victim of a vast fraud revealed in a 2010 U.S. Securities and Exchange Commission suit against Wyly and his brother Charles. The two got rich building businesses including the arts-and-crafts chain Michaels Stores Inc. Charles died in a car crash in 2011.

In 2014, a federal jury in Manhattan found the brothers had used a web of offshore trusts for 13 years to hide stock holdings and evade trading limits, allowing them to rake in $550 million in illegal profit. The verdict quickly triggered bankruptcy filings by Sam and his brother’s widow, Caroline “Dee” Wyly. And then the IRS fight began.

Two-Week Trial

Houser held a two-week trial in January to determine whether the Wylys defrauded the IRS. The proceeding shed light on the assets and lifestyles of the extended Wyly family, including their Dallas mansions, expansive ranch properties in the mountains of Colorado and rare artwork.

The IRS argued many of the luxuries were purchased by offshore trusts and “loaned” to the family to avoid taxes, and that property was gifted to children for the same purpose.

Sam Wyly said he had relied on lawyers and accountants to set up the offshore trusts and knew few details about how they operated. His lawyers called the arrangement “aggressive but not illegal.” Dee Wyly testified that she entrusted financial matters to her husband and signed tax returns and other documents without reading them.

The judge had little patience for Sam Wyly’s defense.

“To accept the Wylys’ explanation requires the court to be satisfied that it is appropriate for extraordinarily wealthy individuals to hire middlemen to do their bidding in order to insulate themselves from wrongdoing so that, when the fraud is ultimately exposed, they have plausible deniability,” Houser said in her 459-page ruling.

Too Complicated

The judge, however, did rule that Dee Wyly didn’t participate in any fraud.

“There is simply no persuasive evidence in the record that Dee understood how these very complicated estate planning transactions worked,” Houser said.

Stewart Thomas, general counsel for the Wylys, said they were pleased with the ruling on Dee, as well as the court’s rejection of an IRS gift-tax claim, but “they are surprised and disagree with the court’s fraud finding as to Sam and his brother Charles.”

The IRS was seeking $1.4 billion from Sam Wyly and $834 million from his sister-in-law, with penalties and interest accounting for 80 percent of the totals, the government said in court papers filed Jan. 25.

The judge gave the parties 30 days to confer and submit a “agreed amounts” on the IRS claims to the court. Failing an agreement, each side is to submit its own proposal, Houser said.

The case is In re Samuel Evans Wyly, 14-35043, U.S. Bankruptcy Court, Northern District of Texas (Dallas).