If rates rise or even level off in 2012, bond funds with more flexibility should have an edge, said Greg Lavine, who owns Eigen's fund.

"We aren't in love with the performance lately, but funds like this have a purpose," said Lavine, an adviser with Altfest Personal Wealth Management in New York, which manages $800 million.

The flexible funds can buy a wide range of fixed-income assets both in the U.S. and around the world and, like hedge funds, they have the freedom to short securities. The funds returned an average of 19 percent in 2009, as riskier assets such as junk bonds and emerging market bonds rallied, said Jacobson. They gained 5.1 percent in 2010, compared with 6.5 percent for the Barclays Aggregate.

This year the funds didn't participate in the Treasury rally. Treasuries returned 8.9 percent through Dec. 5, according to the Bank of America Merrill Lynch U.S. Treasury Master Index.

In an August interview posted on Pimco's website, Chris Dialynas, manager of Pimco Unconstrained, said his fund was positioned to offer "marginal exposure to overall interest-rate risk." The Pimco Unconstrained Fund can have a duration of as low as minus three years, meaning the fund can benefit if interest rates rise. The fund's duration was 2.5 years at the end of October.

Duration measures a bond's sensitivity to changes in rates.

The non-traditional funds have an average duration of about 1 year, compared with 5 years for the Barclays Aggregate Index, according to Morningstar. Treasuries represent 35 percent of the index, according to Vanguard Group Inc.

Pimco Unconstrained had 32 percent of its assets in mortgages, 21 percent in investment grade credit and 28 percent in emerging markets at the end of October, according to information posted on Newport Beach, California-based Pimco's website. The fund had 21 percent in Treasuries, which was more than offset by hedges in the swap and futures markets.

Dialynas, who declined to comment for this story, has said this year that he likes Asian currencies, bonds in selected emerging markets and mortgages, while avoiding Europe. The fund fell 0.1 percent through Dec. 5.

While losing out on the rally in Treasuries, investments the group of funds made in markets such as junk bonds and emerging market currencies haven't worked out as well.