In 2006, Peter Raimondi, a lifelong Bostonian, ripped up roots and relocated to Florida, divorcing from The Colony Group, a highly successful financial advice and tax preparation company he’d founded some 20 years before. Armed with a fat wallet from his stake of the sale and a handful of loyal clients, he claimed a patch of Palm Beach Gardens, a city that had been created overnight by John D. MacArthur in 1960, growing from a population of one squatter to its current population of 50,000 or so. The firm takes its name banyan from one of the city’s most famous symbols, the massive native Indian fig tree whose aerial roots shoot down into the ground, and become hardened wood crutches until the tree has what looks like a number of new trunks.

Raimondi had noticed while visiting his wintering Florida clients over the years something curious: There weren’t many large RIA firms in the area doing business under the banyan trees, despite the number of retirees. (Just behemoth GenSpring in nearby Jupiter.) So the seed of a new kind of banyan had been germinating in the mind of Raimondi, who had been taking business risks since his college days, as he plunked down his own money for a brand new firm—one with which he had much more grandiose plans. Not just an RIA for Florida, but a national firm that made custom investments for picky, moneyed clients with executive concerns.

But by the time he signed the agreement in 2009 for what would be the foundation of his new empire, the market (and assets of his first target firm) had plunged. He had rolled the dice in a bear market.

His solution? Buy anyway. And keep buying.

From that inauspicious beginning, Raimondi went on to buy seven other firms in five years, including, this year, a feather in his cap (or anybody’s)—giant Boston asset manager Silver Bridge, child of the massive national law firm WilmerHale. The purchase doubled the size of Banyan Partners and turned the barely 5-year-old firm into a major player with offices on both coasts. (Since then, he has also announced the purchase of Rushmore Investment Advisors in Plano, Texas, and Holt-Smith Advisors in Madison, Wis., which will raise the AUM to almost $4 billion. Banyan will have 85 employees and nine offices nationwide when the deals close.)

Raimondi also last year hired former Fidelity sales head Scott Dell’Orfano, who brought with him a Rolodex for finding new acquisition targets.

It’s a long way from Raimondi’s beginnings. Most of his undergraduate work wasn’t in business but in film and photography at the Massachusetts College of Art, where he took landscape photos alongside a constellation of 20th century photography stars, such as Harry Callahan, Aaron Siskind and Garry Winogrand. Pictures of Raimondi from back in the day suggest somebody a bit hippieish, says one friend (the only remaining hint is the avuncular beard).

While in school, Raimondi worked for his father’s small security and alarm business with his brothers, doing ladder work and crawling through ceilings. But despite his passion for art, Raimondi wasn’t attracted to the lifestyle, and realized he could serve the art world better as a lawyer. “I actually went to law school to represent artists,” he says. (His older brother, John, is a renowned sculptor.)

While Raimondi was going for his J.D. at Boston University to focus on estate planning, a study group chum who worked at Merrill Lynch one day explained the concept of listed options as the two drove to a job site for Raimondi’s dad. “This was in 1980 and [options] were kind of new and they were a little exciting; for a small amount of money you could control 100 shares of stock. One day I decided to just start tracking options in The Wall Street Journal as I was sitting in the back of contracts class.”

This was the day of big mergers, particularly in the oil and mining businesses, he says. He started charting volume and looking at the spikes. At the time, those derivatives fell outside the scope of the SEC. “I could see all this sort of insider trading activity developing through watching the volume spikes. And I started trading options with a couple of hundred bucks of my own money and a few hundred bucks of my dad’s money.”

He ended up paying his tuition for the next three years this way, and later wrote his thesis in law school on the options market.

He always had work in his father’s business but Raimondi père also encouraged his sons to do more, including start their own businesses. Raimondi did stints with executive financial counseling businesses such as AYCO in Albany, N.Y., and another firm, but he didn’t like the focus on product. So in 1986, he founded a financial planning firm at his dining room table with about nine clients. Working on the AYCO model of executive counseling, he started doing people’s tax returns on an original Macintosh computer and a printer with $4,000 of bonus money.

The new firm grew into the Colony Group and within the year moved from the dinner table to an office on the Boston waterfront, gathering support people and investors along the way. (He started his law firm at the same time.)

“Twenty-eight years ago, there weren’t as many regulations,” he says. “Doing tax returns was really a manual process. There was hardly any software.”
That provided him with the opportunity to advise high-net-worth business executives. “What you had was the ability to manually produce a rather difficult tax return, so you understood every number, where it went and how it correlated with each other,” he recalls. “You became very, very good at giving tax advice. You became very good at dealing with the CEO of a public company in Massachusetts.”

One of his early clients was Ed Eskandarian, the former CEO of Arnold Worldwide advertising company and former minority stakeholder in the Boston Red Sox, who has been a client of Raimondi’s since 1988.  Eskandarian came to trust Raimondi first as a client (Raimondi promised he would save Eskandarian more in taxes than he paid in fees) and then as a co-investor in Banyan. Among other things, the advisor urged caution when Eskandarian was approached by a Canadian company about selling Arnold, one of the giant ad agencies he led, in a complex stock deal.

“I talked to a guy and he made a proposal to me,” says Eskandarian. “Now if I had done that, Peter would have gotten a lot of money to manage, and he told me: ‘Don’t do it. It’s not a good deal for you, and you can do better.’ So I turned down the guy and kept on running the company and sure enough, two years later, I sold it for significantly more.”

Colony expanded over 20 years into a firm with more than $2 billion under management, 55 employees, 650 clients and 14 corporations engaging the firm to do financial counseling work for their top executives. (Today, Colony is part of Focus Financial Partners.) Raimondi oversaw it for many years as CEO, but he had bigger plans, and eventually that led to some disagreements with his partners about the direction of the company. As he saw it, the firm was doing too much low-margin, time-consuming work on tax returns, work whose costs were ballooning. It was the kind of stuff he thought would have been better outsourced to a CPA firm. It held the firm back from being a major international player in asset management, he thought. After a while, talking to his partners “was like talking to a wall,” he says.

“Where it became somewhat disjointed was the combination of having an RIA, an investment management firm, managing close to a billion dollars with its own investment team and then having a rather large financial planning firm doing about a thousand tax returns a year,” he says. “The identity of the firm started to get a little bit confusing.” Raimondi’s intimates add that his partners were more conservative than he was, less willing to take the kind of big risk that Banyan would become.

Raimondi cashed out and went with his wife down south in 2006 to consult for a while and pull an ADV for Banyan. He needed a base to start with—something with a research, investment and compliance apparatus already in place. He first found Oaktree Asset Management in New York, a firm with $250 million in assets and an owner in his 70s, David Bottoms. The latter CEO was also a lawyer with trust and estates experience, but had no real succession plan. (Bottoms is now a managing director at Banyan.)

“So it’s the middle of 2008 and the market is starting to crater, and I’m standing there ready to write a check and ready to take over the firm,” says Raimondi. Realizing he might be about to take a bath, Raimondi took the risk and put down the money anyway. He says Bottoms might have even understood if he’d backed out.

“During that time frame, from the summer of ’08 to when we closed in the first of ’09, [Oaktree’s] assets dropped 20%. Like December 30, we were signing the papers and I looked at him and I said this is either the worst time in the world to buy an investment firm or it’s the best time to buy one.’”

Banyan grew organically for the first couple of years, doubling in assets, while garnering 15 staffers. In 2011, Raimondi bought three more firms, including Weiss Capital Management (the RIA division of a financial publishing company whose publications offered tips on things like short sales to clients scattered across the country; it happened to be right down the street in Palm Beach Gardens) and Colonial Wealth Management (a Boston LPL affiliate whose owner wanted less of a canned investment selection). Those purchases raised Banyan’s AUM to $1 billion. At the time, Raimondi told Financial Advisor, many smaller firms had hit their growth ceilings. That and the recession offered a perfect feeding ground for a hungry acquirer.

“I think probably the most unfortunate thing about the small RIA business—say firms under $600 million—is that they generally need every person they have working for them,” Raimondi said at the time. “So it’s hard to cut staff, which is going to be the largest expense. Most firms are running 50% or above in terms of payroll costs.”

Nowadays, Banyan turns away far more offers than it takes—and there are many firms that want to be wooed. That’s one of the reasons Raimondi tapped Fidelity’s Dell’Orfano, who had worked in the RIA space for 22 years. “Peter respected the fact that I knew a lot of firms,” Dell’Orfano says.

Silver Lining
Dell’Orfano’s hiring might have put the industry on notice about Banyan’s ambitions, but Banyan’s biggest coup was yet to come—its dramatic announcement that it would buy Boston’s Silver Bridge, the RIA arm of WilmerHale, and it’s almost $2 billion in assets. A predecessor law firm, the storied Hale & Dorr, founded in 1918, had branched out with an RIA in 1988 so it could expand investment management to its non-law clients (as The Boston Globe reported, the city has a long tradition of investors turning money over to lawyers, a practice going back to the region’s sea merchant days).

But the ties between the RIA and law sides of WilmerHale had frayed over time, says Tom Manning, the interim CEO of Silver Bridge and now Banyan’s CIO. The customers became different, the growth objectives more disparate. The name had changed to Silver Bridge in 2008 to mark the difference, and the RIA staff found a new building. But despite its level of assets, Silver Bridge was only a drop in the bucket of WilmerHale’s $1 billion in annual revenues, reportedly bringing in only 1% to 2%. The cultural differences between the law and financial sides of the business, coupled with new Dodd-Frank regulations, started to make Silver Bridge a likely spinoff.

Apparently, the wait was too much for some employees, including four partners in the private client group who jumped to rival law firm Choate Hall & Stewart late last year. “We were in the process of having discussions about greater independence with the law firm, but nothing was ever guaranteed that we were going to be able to move in that direction,” says Manning. “And so as opportunities came up for individuals, in certain instances they took those.”

But the loss of some key staff didn’t make Silver Bridge any less attractive to suitors. Manning says the firm met with more than 20. Most weren’t the right fit until Banyan came along. He says Silver Bridge’s proprietary equity strategy and fixed-income expertise meshed well with Banyan’s offerings, including its covered-call expertise and its U.S. large-cap stock acumen. Their philosophies were also similar; both firms were vehemently against the prevalent attitude toward investment management as a mere commodity.

“We actually believe there is an opportunity to brand this organization as an investment-centric firm,” Manning says. “Because no one else is trying to do it. They’ve thrown in the towel on that and are trying to brand themselves in another capacity.” 

While Raimondi ponied up the money himself for Banyan’s first few acquisitions, he needed deeper pockets for Silver Bridge. He was approached by private-equity firms, says Dell’Orfano, but in 2012 turned to Canadian firm Temperance Capital, which was not quite private equity and definitely not into roll-ups. The partners seemed happy to sit by and watch the fun.

Banyan is now on an acquisition tear. Dell’Orfano says he had some 20 conversations in July with different firms that could become Banyan targets. Both Holt-Smith and Rushmore have institutional capability, which is where Banyan wants to head.

Again, Dell-Orfano says Banyan is not looking to buy what he calls “lifestyle” firms—advisors who have hit a ceiling at $200 million or so in assets, stopped growing or who are desperate for succession plans. That window has closed. Instead, he says, he’s looking for firms that can leap into Banyan’s infrastructure—firms with two to three partners in their 40s or early 50s who have started in the last decade and want to aid Banyan’s growth.

Still, that raises the question: How does a motley mix of firms help you build a national footprint and brand?

Manning says Banyan and Silver Bridge both invest for individuals and share a tax-sensitive approach. Banyan boasts internal proprietary strategies with custom portfolios for high-touch relationships with its executive clients. Silver Bridge has approached it from more of diversified asset-allocation process, adding proprietary equity and, especially, fixed-income strategies.

Raimondi says the focus is client service, something he learned from his dad (whom he calls his hero). That means not only will the deep bench of investment strategies be the same, but the client experience should also be the same, whether it’s somebody walking into the firm’s New York, San Francisco, Miami or Boston offices.

Despite all the acquisition activity, Raimondi says that his real ambition is organic growth. He says that the firm has already grown 30% organically this year (as of August 2, before the closing of the Silver Bridge deal). He credits the active sales teams in several cities and the strong referral programs of custodians TD Ameritrade and Fidelity. The covered-call and options business is one selling point. The custom portfolios are a bigger one.

“Every client is really receiving from us a customized response to what they had in mind when they showed up, and that’s not customary in our business because it’s an expensive model,” he says. “You can’t scale that very easily. You have to have a lot of research analysts and a lot of portfolio managers, and that’s expensive. The biggest cost in our business is always going to be employees. The fewer employees you have, the more profitable you can be. … But I just believe that there’s a lot more to this business when you’re trying to manage someone’s life savings.”

So how big does he want the firm to get? The next big milestone would likely be $10 billion in assets. “I don’t think there is a size I would aspire to,” Raimondi says. We’ll get as big as it needs to get to continue to offer all the solutions necessary to prospects. I don’t want to farm out things that I think we can do better.”