Cheap, esoteric and private investments may end up carrying the industry if recent activity is any indication: The biggest inflows this year have gone to long-biased or long-only products run by the quantitative funds, which use computers to decide what to buy and charge lower fees than most.

Renaissance Technologies, for instance, has pulled in $10 billion so far this year, while assets at Two Sigma have risen to $50 billion, up from $38 billion a year ago.

At Luxon Financial, investors have asked for products that can wager on rising and falling prices of securities but that charge much lower fees than hedge funds, said President Anson Beard, whose last job was an executive at a stock hedge fund that closed. Luxon's Cary Street Partners is a $2.5 billion firm of about 40 registered investment advisers whose clients are high-net-worth individuals in the Southeast U.S. and Texas.

Beard said he sees these so-called liquid alternatives taking the place of hedge funds in many investors’ portfolios.

Longford Capital Management raised $500 million for what’s known as a litigation finance fund: It focuses on financing corporate lawsuits and taking a piece of settlements or judgments.

Capstone Investment Advisors, which buys and sells the volatility of stocks, bonds and currencies, has seen assets grow by about $2.5 billion this year on performance and inflows and now manages $5.6 billion.

Family offices, which once seeded hedge funds, are also looking to private deals, whether it’s a swimming pool equipment company or an outfit working on nuclear-fusion technology. John Paulson, who has seen many clients jump ship from his hedge fund after years of poor returns, is trying to lure them back with those that invest in private equity and debt investments.

Even veterans have been getting trounced. John Burbank, whose Passport Capital shot to fame for its lucrative bet against subprime housing ahead of the global financial crisis, told investors in a Dec. 11 letter that he will shutter his flagship hedge fund. He wrote that the fund’s "returns over the past two years are unacceptable and cause me to rethink how to manage money in this environment.''Or consider Lee Ainslie, who started Maverick Capital in 1993 after he left Tiger Management, where he trained with famed investor Julian Robertson. He lost about 10 percent last year and is down again this year.

Traditional stock hedge-fund managers remain hopeful. Ainslie told investors that he expects short selling to again be a profitable pursuit.

“On the short side, periods of frustration are not uncommon and are typically followed by periods in which short selling is actually quite rewarding,” Ainslie wrote in a letter this summer.