LOS ANGELES – Nearly four years after Washington bailed out Wall Street, small banks have yet to repay $11 billion of taxpayer money.

Uncle Sam wants out and is threatening to unload its stakes in the banks at big discounts to new investors. Many of the 324 institutions, mostly tiny community banks and niche players, wonder whether they’ll be able to stay in business.

Some stragglers would become financially unstable if they repaid their part of the $245 billion doled out during the financial crisis by the Treasury Department’s Troubled Asset Relief Program.

Critics said the government, having moved so fast to bail out the big banks, should be nurturing the smaller ones.

“Many of these banks feel forced into a situation they can’t control — a potential fire sale with new and perhaps unfriendly shareholders,” said Jerry Comizio, a banking lawyer at Paul Hastings in Washington. “For some banks, though, given their small size and financial condition, they may not have any other viable option.”

The biggest U.S. banks, four of which have more than $1 trillion in assets each, were able to repay TARP in short order. The fact that so many small banks — most with less than $1 billion in assets each — cannot do so illustrates how the struggling economy has crushed business strategies that often relied on housing-related loans and small-business mortgages.

The Treasury Department bought preferred stock in the banks on the condition that they compensate the government by paying a 5 percent dividend for five years, which then jumps to 9 percent late next year or in 2014. Counting the dividend payments, the government has turned a $19 billion profit overall on its bank bailouts.

Nonetheless, nearly half the 707 banks that received TARP funding have yet to repay their debt, including 164 that at last count had missed one or more dividend payments because regulators have restricted their operations.

The Treasury Department is encouraging the banks to buy back their own stock, which some are planning to do. But most are required to clean up soured loan portfolios, strengthen management and raise new capital privately before regulators will allow them to buy their own stock or even to pay dividends.

Bankers in California said they were struggling to find cash from friendly new investors at prices that wouldn’t wipe out the existing shareholders, often people with deep ties to their communities. The government has signaled that it hopes to dump all its investments within 18 months.

“There’s no capital out there for little banks, no capital for inner-city banks and no capital for banks with a lot of troubled assets,” said Wayne Kent-Bradshaw, an executive with Broadway Financial Corp. in Los Angeles.

The three-branch bank, founded in 1947 by African-Americans to help blacks who had been shut out by mainstream banks, “is all of those,” Bradshaw said.

Some key observers fret over the prospect of investors aiming to break up or resell the banks at a quick profit.

“A key part of our recovery is getting community banks back on their feet,” said Christy Romero, the government’s special inspector general for TARP. She suggested that the Treasury Department consider extending its support to at least some small banks, perhaps by not raising the dividend after five years.

Exiting TARP has been profitable for some, such as Wilshire Bancorp in Los Angeles. The bank was the winning bidder in March when the Treasury Department auctioned its $60 million in Wilshire shares, buying them back for $56.6 million, about 94 cents on the dollar.

“It was a very good opportunity for us,” said Alex Ko, Wilshire’s chief financial officer.