While the U.S. economy appeared to be improving, the bank's policymakers "decided to await more evidence that progress will be sustained" before deciding to slow bond purchases. The bank also cut its full-year economic outlook for this year and the next.
Stock traders shrugged off the Fed's dimmer economic outlook and focused on the continued stimulus, a big reason for the market's bull run over the last 4-1/2 years.
The S&P 500 surged up 20 points, or 1.2 percent, to 1,725 in afternoon trading, having sliced through its previous all-time high of 1,709.67 set on Aug. 2.
The Dow Jones industrial average jumped 153 points, or 1 percent, to 15,683, also above its previous record high of 15,658, also set Aug. 2.
Some investors advised caution, even with stocks hitting all-time highs.
While the Fed's decision is positive for the market in the short term, "investors need to take a step back and consider the idea that maybe the U.S economy is on weaker footing than we originally thought," said Marc Doss, regional chief investment officer for Wells Fargo Private Bank.
Bond prices also rose sharply, sending yields lower. The yield on the 10-year Treasury note fell to 2.68 percent from 2.87 percent a minute before the Fed released its statement a push into bonds by investors not seen since October 2011. That yield is a benchmark for many kinds of lending rates, including home mortgages.
The price of gold jumped $55, or 4 percent, to $1,364 an ounce.
The fate of the Fed's economic stimulus program has been the biggest question on Wall Street for months. It was widely expected that the Fed would cut back on its bond buying at its September meeting.
Tom di Galoma, a bond trader at ED&F Man Capital, said he was "completely shocked" that the Fed decided to wait
In June, Fed Chairman Ben Bernanke laid out a plan to start easing up on the bond-purchase program, and pledged to end it by the middle of 2014, if the economy continued to improve. The Fed's next meeting is October 29-30.
Wells Fargo's Wells and other investors said the Fed might be waiting to see what happens in Washington, D.C. in the coming weeks. A debate over the debt ceiling and the showdown between Congressional Republicans and the White House over the budget looms.
Matt Tom, head of public fixed income at ING U.S. Investment Management, said the Fed's decision to keep the stimulus unchanged likely happened because Bernanke wanted to make sure the economy was ready to function without its help.
Cutting back before the economy was ready would have been much more destabilizing to the market, he said.