Hold on to your handbags.

If spending on bling watches and "it" bags depends on stable markets and happy shoppers, then now is not the time to bet on sellers of luxury goods. Economic uncertainty stretching from China to Europe, along with global stock market and currency swings, have pricked the confidence of high-income shoppers in the U.S. That doesn't bode well for luxury demand.

Even before Britain's vote to leave the E.U. stoked worries over European retailers, China's attempt to drive down conspicuous consumption amid a slowing economy had spooked global brands such as Tiffany and Prada. That, along with a strong U.S. dollar, led to fewer visits from foreign tourists to U.S. department stores such as Neiman Marcus and Bloomingdale's.

Meanwhile, shares in global luxury-goods makers followed by Bloomberg Intelligence have dropped by 13 percent in the past year, compared to a 4 percent fall in the MSCI World Index and a 3 percent increase in the S&P 500.

A slowdown in the U.S., the world's biggest luxury market, could send luxury sellers into a tailspin. Even though robust job gains propped up the U.S. stock market Friday, the long-term trend of slowing job growth, uncertainty over when the Federal Reserve will raise interest rates, and rocky global markets continue to strain the confidence of big spenders who are increasingly choosing to stay on the spending sidelines.

Americans have contributed about a third of the growth in sales of luxury goods in the past three years, second only to the Chinese, according to analysts at Exane BNP Paribas. But now, folks here also seem to be pulling back. 

Its not just weaker tourism. Richemont, which owns brands such as Cartier and Panerai, reckons uncertainty ahead of the U.S. election is also dampening down the once-strong market. Slumping energy prices are also hurting Neiman Marcus, many of whose shoppers' wealth is tied to the oil and gas business.

With the U.S. stumbling, there are few other places for global luxury sellers to find growth. Even Japan, which has been a magnet for Chinese shoppers seeking to take advantage of a weaker yen, is under threat, as the Japanese currency has strengthened against the yuan.

Ironically, one bright spot could be Britain. The pound's weakness against the U.S. dollar and the euro makes it more attractive for overseas visitors, who can get more Hermes scarf or Burberry trench coat for their money. Average spending per transaction by Chinese travelers to the U.K. rose 14 percent in the ten days after the referendum, according to Global Blue, which tracks tax-free shopping. 

In luxury terms, the U.K. remains small, accounting for just 6 percent of the global luxury market, according to analysts at Bernstein.