Long before the phrase "robo-advisor" existed, Sunnyvale, Calif.-based Financial Engines was cornering the untapped 401(k) plan market with computer simulations to help people save for retirement.

With the first wave of baby boomers now retiring, the company is reaching a critical moment: Its core customers, who have been accessing its advice through company 401(k) plans, will soon be off on their own. To hang onto those investors -- and find others -- Financial Engines is planning to go up against other so-called "robo-advisors," companies that use software programs to provide financial advice, and offer advice outside the 401(k) market.

"We do see substantial opportunities beyond the workplace to help investors," Christopher Jones, chief investment officer, told Reuters. "Could we end up competing against the robo-advisors? Absolutely."

With this challenge in mind, Financial Engines, which serves corporate retirement plans from companies including Delta Air Lines to Microsoft to Xerox, is discussing ways to offer retirement savings advice and management to the families and friends of the 8.9 million employees at the companies it serves, Jones said.

Concerns about the low enrollment rates in Financial Engines' money-management service, along with rising costs helped push shares down 45 percent in 2014.

"They are clearly the 800-pound gorilla in the 401(k) space, but the question is, how do they get more people to use the service?" said Martin Schmidt of H2Solutions, a Wheaton, Ill.-based consultant for 401(k) plans. "As the baby boomers retire, that is going to be a bigger challenge."

Meanwhile, a growing number of robo-advisor upstarts like WealthFront and Betterment, as well as more traditional brokerage firms, such as Charles Schwab Corp., are spending heavily to convince soon-to-be retirees to move their retirement savings into individual retirement accounts with them. In 2013, investors rolled $324 billion from 401(k) accounts into the $6.5 trillion IRA market, according to Cerulli.