“You can land a seaplane or a huge yacht on the dock,” Devendorf says. “And you can put a boathouse anywhere on the island.”

The Next Owners

As time went on, Morgan apparently found the house isolated enough that—starting in the 1940s—he allowed “various owners and friends to build their own residences on the island,” Devendorf explaine. “Each of the houses had five or six acres surrounding them.”

After President John F. Kennedy’s assassination, Jackie Kennedy rented one of those houses for a summer while brother-in-law Robert F. Kennedy rented an estate nearby.

Morgan died in 1960; the house remained in his wife’s possession until the 1970s, when it and 19 acres of grounds were sold to a Texas-born coal magnate named John Samuels.

Samuels had made waves in New York’s charity circuit in the 1970s, donating enough to become chairman of City Center, New York City Ballet, and City Opera. Starting in the early 1980s, though, Samuels’s businesses began to crumble, and by the ‘90s he’d filed for bankruptcy. A New York Times article from 1993 details the public auction of the estate: The minimum bid was apparently $5.7 million; before the sale, qualified bidders had to hand the broker a certified $350,000 check.

The buyer was Margo Walker, described in a 1997 Vanity Fair story as “a North Shore socialite and real-estate broker.” (Devendorf, the broker, declined to comment on the owner.) Walker was reportedly in a relationship with Michel David-Weill, former chairman of Lazard Frères, according to The Last Tycoons, William D. Cohan’s inside look at the investment bank. “With Michel’s help, she had already purchased three of the five houses” on the island, Cohan recounted.

Not Out of the Ballpark

Over time, Walker managed to buy the rest of the homes on the island, which include the graceful, stone-walled, four-bedroom “Creek House,” the substantial, stucco “Palm-Beach-style” Pond House, which also has four bedrooms, and the nameless Palladian, waterfront manor house, which has six bedrooms. The island also has three “cottages,” one of which has four bedrooms, another of which has two, and the third of which has one bedroom and one bath.

The island also comes with horse stables, paddocks, a “groomsman’s cottage,” and an original garage that can house eight cars. In total, then, omitting servant’s quarters, the estate has at least 28 bedrooms and sleeps nearly double that amount.