Former Federal Reserve chairman Ben Bernanke on Tuesday blamed Congressional inaction for prolonging the Fed’s low interest rate policy.

He also dismissed concerns about inflationary effects from unwinding the Fed’s balance sheet.

Bernanke, who was greeted warmly by advisors attending TD Ameritrade Institutional’s Elite conference in Dana Point, Calif., spent an hour and a half at the event reliving the Fed’s response to the financial crisis and taking questions from TD Ameritrade Institutional managing director Jim Dario.

Given the length and severity of the recession, there’s a question of whether the Fed should have done more, Bernanke said.

But “the country has put too much on the shoulders of the central bank,” he said. “And it would have been better if there had been a more balanced program that included [more] fiscal action, and things to improve the function of the labor market [and] the housing market.”

With a more balanced approach, the Fed could have raised rates, he said.

“The reason interest rates are so low is that everyone has put all the weight on the Fed to do the work,” Bernanke said. “We’ve seen this in other countries as well.”

The recent budget battles in Congress have also delayed a rise in rates, he said.

With the sequester and other spending cuts, “fiscal policy was quite tight in 2013,” Bernanke said, “which didn’t do much to help long-term fiscal stability because it was all focused on the very near term. … What it did do is push against what the Fed was trying to do” with its accommodative policies.

“The [Congressional Budget Office] estimated that federal fiscal policy in 2013 cost the economy about a point-and-a-half of growth,” he said. “So instead of 2 or 2.5 percent [GDP growth] we could have gotten 3.5 or 4 percent, and we could have had that with higher interest rates.”

The showdown in Washington over the budget ceiling “was a pure loss” to the economy, Bernanke added. “All it did was create uncertainty and hurt confidence,” he said.

The political dysfunction hurt during the crisis as well. “There was a chance that the system, not just the U.S. system but the whole global financial system, [could] have collapsed,” he said. At the time, “I said to my colleagues, ‘There are no atheists in foxholes and no laissez-faire policymakers in a financial crisis,’ but I was wrong. It took two tries to get Congress to take any action” on a bailout plan. “Our political system had a lot of trouble ... taking action that was needed in a timely way.”

About the risk that easy money will create more bubbles, Bernanke said the response should be for the “regulatory system to look for excessive risk taking” so that monetary policy is unaffected.

Bernanke dismissed concerns about the size of the Fed’s $4 trillion balance sheet, which has ballooned with its asset purchases. That $4 trillion is not large relative to the size of the U.S. economy—and about normal compared to other central banks, he said.

“For that reason and other reasons, I don’t think it’s necessary to shrink the balance sheet. If [the Fed] decides to do so, it will do so over time, by letting securities run off as they mature and not replacing them. If they go that way, it will take a number of years to get back to the customary level,” he said.

“But the key thing to understand is, even if the balance sheet stays where it is or gets bigger, it doesn’t prevent the Fed from tightening monetary policy and raising interest rates at the appropriate time, so there’s no inflationary risk or anything like that,” he said.

The Fed has plenty of tools to raise interest rates, Bernanke said, such as paying interest on reserves that banks hold at the Fed, or “an even more powerful … new tool the Fed has called a reverse repurchase agreement [that] would allow any private sector institutional investor … to make short-term deposits at the Fed at an interest rate the Fed sets. What that would do, of course, is set a floor on money market rates” by establishing a risk-free rate for any market participant.

Dario asked Bernanke if, in hindsight, there was anything he would have done differently.

“You mean besides not taking the job?” Bernanke quipped.

“Obviously, if we’d had a better appreciation of the vulnerability of the system to the mortgage markets … we would have done more I think to demand more capital and demand more caution in lending,” he said.

And the Fed “was somewhat optimistic about the pace of the recovery” in its early stages and perhaps should have done more to stimulate the economy, he said.

Bernanke, a baseball fan, has been mentioned as a possible new head of Major League Baseball.

Would he take the job? Dario asked.

“I’m waiting for the call,” Bernanke joked. “I’m not counting on that particular direction.”