Massachusetts' pension fund is pushing to move some of the $5.6 billion it invests with hedge funds into separately managed accounts and may even invest, at a lower cost, in strategies designed to mimic hedge fund returns.

The move is part of a broader effort by institutional investors to push back against hedge funds' high fees at a time returns have been lackluster. But few big investors have been as public about their plans as Massachusetts.

The state's Pension Reserves Investment Management Board, known as PRIM, invests 9.5 percent of its $59 billion with some of the industry's biggest names, including Larry Robbins, William Ackman, Paul Singer, Daniel Och and Andreas Halvorsen.

Now the pension fund is asking them and other managers to segregate the state's money in so-called separately managed accounts, which would potentially allow the state to negotiate better terms with managers and keep closer tabs on its cash.

"That's the next frontier and we might be at the forefront of it," said Michael Trotsky, a former hedge fund manager who has been the pension fund's executive director since 2010. "It will give us full transparency and the ability to better control the assets," Trotsky, who is also the fund's chief investment officer, said in an interview.

Trotsky said the pension fund could also start to allocate money to hedge fund replication strategies that seek to mimic their returns but for less than traditional fees, which often include a 20 percent performance fee on top of a 2 percent management fee.

Under Trotsky's guidance, the state has already shown its penchant for cutting costs when it eliminated an entire layer of fees by firing its funds of funds managers and making direct investments with 23 hedge funds. That saved the fund $29 million in fees last year alone, Trotsky said.

"I value a basis point of cost reduction more highly than a basis point of return," he said, adding, "We don't know from year to year what the markets will return but we can predict with a lot of accuracy what our cost structure will be."

Now he is ramping up the push into separately managed accounts by getting ready to hire a managed account provider who will provide risk and transparency reporting and monitor the accounts. He expects the trustees to vote on the candidate in August after they voted this week to hire Arden Asset Management as its hedge fund investment adviser.

To be sure, Massachusetts is happy with its current lineup of 23 hedge funds and has no plans to cut anyone, Trotsky said. He said he may even increase the lineup to 25 to 30 firms.

Pension funds have held separately managed stock and bond accounts with mutual funds for years. But more exclusive hedge funds have been reluctant to go this route, arguing that any special deal with one client could prompt others to ask for the same thing and that this kind of tailoring could cause them administrative headaches.

"I'd characterize it as a conversation that very few (hedge funds) are willing to have," Trotsky said. The individual hedge funds declined to comment.

Over time though, Trotsky hopes that hedge funds see the creation of separately managed accounts as a win-win proposition where the state and manager can work more collaboratively.

For example, if the pension fund had a managed account with a skillful manager who is sitting on a pile of cash, the state might suggest that the manager take on a little more risk and put more money to work. "Managed accounts allow us to tailor the program and better meet our needs," Trotsky said.