Fidelity Investments is pulling the $500 million it has invested with Ken Fisher’s investment firm, according to people familiar with the matter.

The move by the mutual fund giant is part of a growing exodus of big clients in the wake of vulgar comments made by Fisher at an industry conference. Fidelity’s divestment brings the total yanked from Fisher Investments to more than $1.8 billion.

Fisher Investments is seeing an intensifying backlash since the firm’s founder made offensive comments about women, spoke of genitalia and then failed to immediately understand the gravity of his words. Fidelity joins pensions in Iowa, Michigan, Boston and Philadelphia that have divested their money.

A spokesman for Fidelity declined to comment.

Camas, Washington-based Fisher, which managed $112 billion as of September, is also facing scrutiny from several other pensions. A representative for Fisher couldn’t immediately be reached for comment.

Fisher Investments was managing about $10.9 billion on behalf of 36 state or municipal government entities, including pension plans, at the end of 2018, according to the firm’s Securities and Exchange Commission registration. That figure is down from $13.2 billion at the end of 2017.

At the Tiburon CEO Summit on Oct. 8, Fisher compared the process of gaining a client’s trust to “trying to get into a girl’s pants” and talked about genitalia. Fisher has apologized for the comments.

--With assistance from Miles Weiss, John Gittelsohn, Sridhar Natarajan, Annie Massa and Janet Lorin.

This article was provided by Bloomberg News.