Every country has its own way of taxing income, savings, investments, and pensions, sometimes making it impossible to explain to the IRS what’s going on. Tax treaties can help specialists navigate some of these issues, but these agreements can be enormously complicated and maddeningly vague. Limmer said that even in the U.K.—where “at least we share a common language”—accountants can disagree with each other on how to sort things out, especially in the “particularly tricky” area of pensions.

Milburn isn't an accountant or lawyer, but she has noticed the same thing: “I actually don’t think that the IRS or tax professionals necessarily know 100 percent what to do. I think it’s always a bit of a gray area. Because how can you convert something in another country to a U.S. equivalent?”

IRS agents stationed at U.S. embassies used to be able to help, but budget cuts forced the agency to close the last of those offices (in London, Paris, and Frankfurt) last year.

Double Taxation
Because the rules are so confusing, some say they often end up being taxed unfairly, paying the IRS and their home country on the same income. Brian Krahmer, 40, a Minnesota native who moved to Germany in 2014, must pay U.S. self-employment tax on his freelance income–even though the work, mostly software development, is for German companies. “If I’m already filing a German income tax return on the money earned, I don’t see any fairness from also having to file in the U.S.,” he said.

The rules can feel unfair, even when they don’t technically result in double taxation. For example, the IRS demands that Americans pay capital gains taxes on sales of homes in the U.K.—gains that can be greatly inflated by currency swings. The U.K. doesn’t have the same tax, but it does impose a tax on home purchases. An American in London who wants to move has the pleasure of paying both.

Treated Like a Criminal
The IRS’s fight against tax evasion has had its successes. Many hidden Swiss bank accounts are no longer so secret, for example. But provisions that catch millionaires hiding money overseas can also ensnare middle-income Americans working and living abroad. As a result, banks and investment companies, forced by the IRS to keep close track of their American clients, are becoming reluctant to take them on.

“We, as Americans overseas, cannot live normal lives,” said Redmond, originally from Washington, D.C. “We’re seriously limited in being able to save like stateside Americans.”

Most living outside the U.S. simply want to know how much they owe the IRS vs. the local tax collector. “These are not people who are hiding money,” Limmer said. Nine times out of 10, she estimated, an American living in London is paying more tax than a comparably compensated British citizen.

Passport Threat
The new passport-revocation rule, slipped into a transportation funding bill signed by President Barack Obama, raises the stakes. It allows the U.S. to revoke the passport of any American whose tax debt exceeds $50,000.

It’s not hard to see how expatriate taxpayers could get to this level, especially if they’re late in realizing they needed to file in the first place. The fines for failing to report bank accounts are high; the IRS can impose a penalty of $10,000 for each violation of the rules. “If the government enforces this in the most stringent way possible, this could be hugely horrible for people who live overseas,” McKeegan said.