A federal judge in Texas ruled that former billionaire Sam Wyly defrauded the IRS by shuffling assets among a network of offshore trusts to evade millions of dollars in taxes.

U.S. Bankruptcy Judge Barbara Houser in Dallas said Tuesday that Wyly, a “sophisticated” businessman, was aware of the efforts of his advisers to hide his assets. Houser roundly rejected arguments that the 81-year-old entrepreneur was simply following orders from his own employees.

“The court does not believe that the law permits Sam to hide behind others and claim not to have known what was going on around him,” she said.

The ruling is the latest blow to Wyly, once a fixture in Texas high society, in a string of legal battles with U.S. agencies that threatens to wipe out the fortune he amassed over a lifetime.

The Internal Revenue Service claimed to be the victim of a vast fraud revealed in a 2010 U.S. Securities and Exchange Commission suit against Wyly and his brother Charles. The two got rich building businesses including the arts-and-crafts chain Michaels Stores Inc. Charles died in a car crash in 2011.

In 2014, a federal jury in Manhattan found the brothers had used a web of offshore trusts for 13 years to hide stock holdings and evade trading limits, allowing them to rake in $550 million in illegal profit. The verdict quickly triggered bankruptcy filings by Sam and his brother’s widow, Caroline “Dee” Wyly. And then the IRS fight began.

Two-Week Trial

Houser held a two-week trial in January to determine whether the Wylys defrauded the IRS. The proceeding shed light on the assets and lifestyles of the extended Wyly family, including their Dallas mansions, expansive ranch properties in the mountains of Colorado and rare artwork.

The IRS argued many of the luxuries were purchased by offshore trusts and “loaned” to the family to avoid taxes, and that property was gifted to children for the same purpose.

Sam Wyly said he had relied on lawyers and accountants to set up the offshore trusts and knew few details about how they operated. His lawyers called the arrangement “aggressive but not illegal.” Dee Wyly testified that she entrusted financial matters to her husband and signed tax returns and other documents without reading them.

The judge had little patience for Sam Wyly’s defense.