WASHINGTON — Fake IRS agents have targeted more than 366,000 people with harassing phone calls demanding payments and threatening jail in the largest scam of its kind in the history of the agency, a federal investigator said Thursday.

More than 3,000 people have fallen for the ruse since 2013, said Timothy Camus, a Treasury deputy inspector general for tax administration. They were conned out of a total of $15.5 million.

The scam has claimed victims in almost every state, Camus said. One unidentified victim lost more than $500,000.

“The criminals do not discriminate. They are calling people everywhere, of all income levels and backgrounds,” Camus told the Senate Finance Committee at a hearing. “The callers often warned the victims that if they hung up, local police would come to their homes to arrest them.”

The scam is so widespread that investigators believe there is more than one group of perpetrators, including some overseas.

Camus said even he received a call from one of the scammers at his home on a Saturday. He said he had a stern message for the caller: “Your day will come.”

Sen. Johnny Isakson, R-Ga., said he got a similar call, but realized it wasn’t a real IRS agent.

“It was a very convincing, convincing phone call,” Isakson said.

So far, two people in Florida have been arrested, Camus said. They were accused of being part of a scam that involved people in call centers in India contacting U.S. taxpayers and pretending to be IRS agents.

“These criminal acts are perpetrated by thieves hiding behind telephone lines and computers, preying on honest taxpayers and robbing the Treasury of tens of billions of dollars every year,” said Sen. Orrin Hatch, R-Utah, chairman of the Senate Finance Committee. “Taxpayers must be more aware of the risks and better protected from attack and these criminals must be found and brought to justice.”

The IRS and the inspector general’s office started warning taxpayers about the scam a year ago, and it has since ballooned. This year, it tops the IRS list of “Dirty Dozen” tax scams.

Tax scams often increase during tax filing season, when millions of Americans are preparing their returns.