The richest colleges performed worse than small endowments in the most recent year.
Graduate students for the first time are set to outpace how much undergraduates are borrowing.
Colleges needed an annual gain of 8.2% last year to keep pace with inflation and spending.
If the Education Department was a bank, it would rank fifth by size of assets in the U.S.
Fees on schools with big endowments would be used to fund community colleges.
The returns contrasted sharply with the previous year's results, when endowments returned a median 27%.
Strong endowment returns have helped bolster schools' balance sheets, driving the borrowing surge.
For decades, tuition has risen faster than prices, making college costs a flashpoint.
Advocates who have been pushing for student debt to be canceled say temporary relief doesn't go far enough.
A number of colleges have hit the investment income threshold that triggers a federal tax.