Brown Brothers Harriman has agreed to pay a record $8 million civil fine for "substantial" violations involving its program to detect and prevent money laundering, Wall Street's industry- funded watchdog said today.

The New York-based investment firm did not have an adequate program in place to look for and detect suspicious penny stock transactions, the Financial Industry Regulatory Authority (FINRA) said in a statement. The firm also did not sufficiently investigate potentially suspicious activity involving penny stocks after becoming aware of a possible problem, nor did it file mandatory reports to alert regulators, FINRA said.

The fine is the highest levied by FINRA for violations of the securities industry's anti-money laundering compliance rules, the regulator said. FINRA also levied a $25,000 fine against the firm's global anti-money laundering compliance officer, Harold Crawford, and suspended him for 30 days.

The transactions in question generated at least $850 million in proceeds for Brown Brothers customers, according to FINRA.

Brown Brothers Harriman and Crawford neither admitted nor denied FINRA's charges, according to FINRA. Brown Brothers has changed its handling and monitoring of low-priced securities to prevent a possible recurrence of problems, the firm said in a statement. The activity involved a "small part" of the firm's investor services business and not its investment management or private banking business, it said.

"I believed that we had established an appropriate system to identify and deal with global AML risks," said Crawford, the firm's anti-money laundering compliance officer, in a statement. FINRA, Crawford said, did not consider certain measures he took, including severing ties with "certain problematic" clients, so he settled, he added.

Low-priced securities, such as penny stocks, are often subject to efforts by fraudsters to falsely inflate trading volume and share prices, a securities law violation that is a precursor to money-laundering, according to anti-money laundering compliance professionals. Penny stocks typically trade at less than one dollar per share and are highly speculative.

The problems at Brown Brothers happened between early 2009 and mid-2013, according to FINRA. The firm executed transactions or delivered securities involving at least six billion shares of penny stocks, many on behalf of foreign bank customers in known bank secrecy havens such as Switzerland and Guernsey, according to its settlement with FINRA. Fraudsters who do business through bank secrecy havens can more easily shield their identity, say compliance professionals.

Brown Brothers also did not know other basic details about the transactions, such as the identity the stock's true owner. The absence of those details should trigger a review of the transactions by a firm's anti-money laundering team, said Aaron Kahler, director of anti-money laundering compliance services for a U.S.-based unit of Capgemini, a Paris-based global consulting firm.

Firms that do not have that information may not be fully aware of the potential for fraud or fail to properly analyze transactions, said Kahler, who was not involved in the case.