The market for luxury homes in the Hamptons, the summer playground for Wall Street's wealthiest, is losing some of its luster as financial markets limp along for a second year.

The average price of the 10 most expensive homes sold in this cluster of towns, villages and hamlets on Long Island's east end was $35.5 million in 2015, 20 percent lower than the $44.6 million recorded the year before, according to real estate brokers Town & Country Real Estate in East Hampton.

That is far from calamitous given it is the second-highest average top 10 price ever and up from just $15.9 million in 2009, the year the market bottomed during the financial crisis.

But for Judi Desiderio, who has been active in Hamptons real estate for three decades and is now Town & Country's chief executive, it is still a meaningful decline.

Her maxim is that record sales prices are only shattered following outstanding years for U.S. stocks and that the Hamptons real estate market goes in cycles in lockstep with Wall Street's fortunes. Excluding dividends, the S&P 500 benchmark stocks index lost 0.7 percent last year and is down about 1 percent so far this year.

The top 10 numbers may be skewed by one or two of the highest priced sales but the softness is also reflected in a wider survey by brokerage Corcoran Group that shows the median price for the most expensive 10 percent of Hamptons sales (57 homes), declined about 4 percent to $7.6 million in the fourth quarter of 2015 from a year earlier.

Desiderio said the next boom may not happen for some years. "We won't see this again until 2021 as it seems to run in seven-year cycles," she predicted.

Wall Street's performance and luxury home prices in the Hamptons are inextricably linked, especially the level of bankers' bonuses as they often finance second homes, said Anthony DeVivio, managing director in the Hamptons for another brokerage, Halstead Property.

The average bonus from a Wall Street bank was likely 5 percent to 10 percent lower in 2015 than the previous year, the first decline since 2011, said Alan Johnson of compensation consulting firm Johnson Associates Inc.

Hedge funds also have struggled, with an average return of just 0.04 percent last year, according to the Barclay Hedge Fund Index. In 2014, it wasn't much better - with a 2.88 percent gain.