NEW YORK – The Wall Street insider trading investigation may lead everyday investors — already rattled by a stock market meltdown, a one-day “flash crash” and the Madoff scandal — to finally conclude that the game is rigged.

“A large part of trading has to do with trust, and I don’t have it,” says Mark Swenson, a 43-year-old plumber from New Hampshire who refuses to buy individual stocks.

“When a stock moves up 10 percent, you don’t know why,” he added. “We can pretend that everyone has access to the same information, but they don’t.”

Even before news broke that federal investigators were looking into whether hedge funds traded on inside information, small-time investors were pulling their money out of stocks — despite a remarkable run for the market since spring 2009.

Americans have pulled $60 billion out of U.S. stock funds this year, according to the Investment Company Institute, a trade group. Meanwhile, investors have piled money into Treasuries and bond funds that are considered safer investments, even if they don’t return as much money. And at the same time, banks like Wells Fargo have reported that money is moving into checking and savings accounts.

To be sure, it’s natural for people worried about their jobs or the falling value of their homes to sock cash into more conservative investments. But this has been no garden-variety recession.

It has coincided with turmoil in the stock market that goes back a decade, to the collapse of the Internet bubble and portfolio-draining scandals involving high-flying companies such as Enron and WorldCom.

More recently, investors have lived through the housing bubble, the collapse of Wall Street firms such as Bear Stearns and Lehman Brothers and the Bernard Madoff scandal.

“Virtually everyone on the Street believes there are significant improprieties, and I think there is an even more important point for the massive number of investors who are not Wall Street players,” says former New York Gov. Eliot Spitzer, once known as the “sheriff of Wall Street” for aggressively prosecuting white-collar crime as state attorney general. “And that is for most of us, you can’t beat these guys at their own game.”

People are nervous about the state of their assets in part because their homes are worth so much less these days, not to mention job insecurity and slow economic growth overall.

Some pros on Wall Street say hesitation by small investors is good news. It means that there’s plenty of “dry powder” to propel the market higher in the next few months when and if the little guy finally relents and joins in the rally.

The insider-trading probe could test that theory. The FBI this week searched offices of three hedge funds. Some of Wall Street’s most influential firms, including Janus Capital Group, have been subpoenaed.

On Wednesday, an employee of a firm that supplied market intelligence to hedge funds was arrested and charged, among other things, with conspiracy to commit securities fraud.

It was not yet known whether the man dealt with the funds raided this week.