Investors’ rekindled passion for sales of private mortgage securities is cooling.

Yields on $397.3 million of top-rated securities sold yesterday by Redwood Trust Inc. were 175 basis points more than benchmark swap rates, up from 170 basis points on bonds it sold earlier this month and 97 basis points, or 0.97 percentage point in January. Sales this year of home-loan bonds without government backing have already surpassed the highest annual total since the debt sparked the global financial crisis.

“Suddenly, in the last couple of months, the love has dissipated” as investors demand to be paid more relative to so-called agency bonds, said Brad Friedlander, a managing partner at Atlanta-based Angel Oak Capital Partners, which oversees about $2.5 billion.

The growing pains come as privately issued bonds begin to recover market share from the government-backed securities that have accounted for more than 90 percent of lending since 2008. The bonds have all been backed by jumbo loans that exceed the limits for taxpayer-supported guarantors Fannie Mae and Freddie Mac, so the primary competition in that market has been banks holding the loans on their books.

Issuance of non-agency mortgage securities so far this year has been tied to $5.2 billion of new loans, up from $3.5 billion last year and less than $700 million in 2011, according to data compiled by Bloomberg. Sales peaked at $1.2 trillion in both 2005 and 2006 before collapsing as their prices tumbled amid soaring foreclosures and plunging real-estate values.

Wider Spreads

Wider spreads on new bonds may slow issuance by making it more attractive for banks to retain jumbo mortgages, according to Credit Suisse Group AG analysts led by Chandrajit Bhattacharya. The firm forecast in a report yesterday that sales may approach $15 billion this year.

The rising relative yields may also postpone the day when private bond issuers will be able to compete with Fannie Mae and Freddie Mac for smaller loans. The two companies have been increasing their guarantee fees as their overseer tries to reduce their role in lending.

Redwood Chief Executive Officer Martin S. Hughes and Chris Katopis, executive director of the Association of Mortgage Investors, said at a Congressional hearing this week that in addition to competition from the government, the private mortgage-bond market is being held back by unresolved issues tied to its collapse, including unfinished regulations, investors’ distrust of lenders and the government, perceived conflicts of interest at loan servicers and the ability of borrowers to reduce their home equity with second loans.

Biggest Issuer

Michael McMahon, a Redwood spokesman, declined to comment on its deal. The company, based in Mill Valley, California, has been the biggest issuer since the market restarted in 2010. This year, it’s responsible for $3.1 billion of the total.

Others sellers have included Credit Suisse, JPMorgan Chase & Co. and EverBank Financial Corp., whose President W. Blake Wilson said on a conference call yesterday that its transaction helped its mortgage-sale margins buck an industry trend by rising last quarter. PennyMac Mortgage Investment Trust, run by former Countrywide Financial Corp. executives, said this week it expects its first deal next quarter.

Scott Simon, who heads mortgage-bond investing at Pacific Investment Management Co., called new non-agency securities “insanely expensive” in February. At the time, spreads had been narrower than similar Fannie Mae bonds, according to Credit Suisse. Now they are paying about 10 basis points more, the bank’s analysts said.

Seasoned Debt

In the sale of bonds last month by EverBank, underwriters Bank of America Corp. and Barclays Plc pushed investors seeking higher yielding, junior tranches to also buy the top-rated AAA portions, people familiar with the offering said. Their effort showed demand is greater for the lower-rated bonds with potentially higher returns.

Friedlander, whose Angel Oak Multi-Strategy Income Fund has beaten 98 percent of mutual-fund rivals this year, said that instead of the top-rated debt, he still prefers “seasoned” securities issued earlier with spreads more than 200 basis points higher because credit ratings aren’t a big focus in his strategy.

The non-agency market’s recovery has been driven by investors seeking higher yields as the Federal Reserve keeps interest rates at record lows. At the same time, Fannie Mae and Freddie Mac have raised their bond-guarantee fees and in 2011, the upper limits for the size of the mortgages they can buy were lowered.

Sluggish Growth

The volumes have been limited by banks seeking loans for their own portfolios amid sluggish economic growth. Banks retained 14 percent of new home loans last year, up from 9.6 percent in 2011, helping to scale back the government’s share, according to Amherst Securities Group LP calculations. In the past three quarters, Wells Fargo & Co. retained $22.9 billion of mortgages with balances small enough to place into Fannie Mae and Freddie Mac bonds, according to its disclosures.

Jumbo mortgages are those larger than allowed in government-supported programs, currently as much as $729,750 for single-family properties in high-cost areas. Limits range from $417,000 to $625,500 for Fannie Mae and Freddie Mac loans with the lowest costs for borrowers using 20 percent down payments.

Fannie Mae and Freddie Mac loans that are more than $417,000 fall into a special category that can’t be traded in the main part of the government-backed bond market, leaving them with higher rates. Redwood’s Hughes said at the April 24 hearing of the House Financial Services Committee that Wells Fargo two days earlier had been offering those “agency jumbo” loans at 3.63 percent, compared with its quotes for true jumbo loans of 3.88 percent.

Higher Yields

Hughes said “we are not that far off” from closing the gap, though the wider spreads on the AAA securities Redwood sold yesterday may make that harder.

Bond investors’ demands for higher yields may already be impacting borrowers, with jumbo loans costing 88 basis points more than smaller mortgages yesterday, up from 42 basis points on March 8, according to data from LoanSifter Inc., a mortgage technology firm whose information has been cited in Fed papers. Still, that’s down from 141 basis points in September.

The non-agency market this year has also wrestled with lenders and bond issuers including JPMorgan and Credit Suisse seeking to build deals with more protections for themselves against forced repurchases of misrepresented debt.

Investor Protections

Moody’s Investors Service said in a March report that the relief granted in JPMorgan’s first deal in the market since 2008 was so extensive that it meant the top ratings competitors granted weren’t appropriate. Rating firms have said Redwood’s deals offered strong investor protections, though larger banks may be better able to survive repurchase demands.

“There has been so much litigation that it’s going to be a painful process to get the wording in place that all sides can live with,” said Michael Canter, head of securitized assets at AllianceBernstein Holding LP, which manages $443 billion.

While Canter said he would accept greater limits on repurchases when a misrepresentation isn’t tied to the reason a borrower defaults, the “language needs to be carefully worded.” He would be concerned if bond contracts didn’t call for all defaulted loans to be reviewed for underwriting mistakes, an investor-friendly shift from the protocols before the crisis. The Redwood deals include that provision.

“I want to see a thriving mortgage market away from the government,” he said. “For that to happen we need originators to be comfortable originating and securitizing in the non-agency market, and investors need to be comfortable, too.”