Not only does CARS probe the devil in the details, it is driving standardization of financial reporting in the industry. It requires standardized GAAP-compliant financials, which it drives through its reporting template. Even so, it has not been able to standardize how CDFIs account for portfolio performance.

"We rate organizations that have loans on their books that are 360 days delinquent and even 1,000 days delinquent," Chapel says. "They haven't written off [these loans]because they know the borrower so well. They're waiting for some event to happen, and they are confident it will happen.

"You look at their overall loan losses, and they're right," she points out. "But it's just not good practice. You mask what the real risk is."

The most complex––and most time-consuming––part of the CARS analysis is assessing the asset quality of each CDFI's portfolio. To gauge the portfolio’s performance over time, it looks at the types of borrowers and loans, as well as delinquencies, restructures and write-offs. It also examines how a CDFI conducts its own due diligence on the loans it makes and how it monitors those loans. And if the CDFI is not 100% deployed in its own loan fund, it checks what those assets are invested in.

Trillium's Rice points out that the purpose of the CARS exercise is transparency, and not to force CDFIs to write off loans.

"If they have a lending profile that's the same as their local community bank, why bother?" he says. "We would rather see a loan fund work with its borrowers to restructure a loan if they can keep it performing rather than write it off. It's better financially, and it's part of the mission. But at the same time, we want the loan fund to measure it and manage it."

Meaningful Datapoints
Although CARS wants CDFIs to measure and manage their social impact, it realizes that standardized data isn’t appropriate because each CDFI has a different mission, Chapel says. Instead, CARS asks each CDFI to identify what datapoints are most meaningful in terms of assessing how well it's achieving its mission, and then to explain how it collects the data and uses it to monitor its own performance.

"We look hard at how the data lines up with what they are trying to do, and their programs and strategies, " Chapel says. "And we go into the systems they use to collect and report the data."

In this case, though, numbers don't tell the whole story and can even be misleading. That's because CDFIs often act as catalytic agents. For example, they may provide high risk gap financing to a business or development project, which can then attract funding from conventional lenders. Or a CDFI may finance a critical development of a major corner of a neighborhood that leverages other development.

"To say that one CDFI financed 3,000 affordable homes versus another that financed 40,000 does not tell you anything because it's out of context," Chapel says. "There are so many groups that are thoughtful about how to monitor their social impact and do a good job. But in our experience, investors are not interested in the impact analysis we do. They care about financial risk. They want to know if they will get their money back."