Obama has been weighing whether a Rice nomination would be worth the fight. He accepted her decision with a shot at Republicans.
"While I deeply regret the unfair and misleading attacks on Susan Rice in recent weeks, her decision demonstrates the strength of her character," he said.
If Obama taps Kerry for State, the president will create a potential problem for Democrats by opening a Senate seat one that recently defeated Republican Sen. Scott Brown is eyeing. Brown had been elected as Massachusetts' other senator in January 2010 after Democrat Ted Kennedy died, stunning the political world as he took the seat held by Kennedy for decades. Brown lost that seat in the November election.
Rice had become the face of the bungled administration account of what happened in Benghazi, Libya, on Sept. 11, 2012 when four Americans, including the U.S. ambassador to Libya, were killed in what is now known to have been a terrorist attack.
Obama had defiantly declared he would chose her for secretary of state regardless of the political criticism, if he wanted, but such a choice could have depleted him of capital and gotten his second term off to a turbulent start with Capitol Hill. Already, Rice's withdrawal underscored Obama's difficulty in pursuing his next agenda in a time of divided and divisive government.
Rice withdrew her name in a letter to Obama and in a media rollout aimed at upholding her reputation.
She said she was convinced the confirmation process would be "lengthy, disruptive and costly to you and to our most pressing national and international priorities."
"That trade-off is simply not worth it to our country," Rice said.
Rice may end up close to Obama's side in another way, as his national security adviser should Tom Donilon move on to another position. The security adviser position would not require Senate confirmation.
Obama made clear she would remain in his inner circle, saying he was grateful she would stay as "our ambassador at the United Nations and a key member of my Cabinet and national security team." Rice, too, said in her letter she would be staying.
She would have faced strong opposition from Senate Republicans who challenged her much-maligned televised comments about the cause of the deadly raid on the U.S. Consulate in Benghazi, Libya.
Her efforts to satisfy Sens. John McCain, Lindsey Graham, Kelly Ayotte and Susan Collins in unusual, private sessions on Capitol Hill fell short. The Republicans emerged from the meetings still expressing doubts about her qualifications.
"The position of secretary of state should never be politicized," Rice said. "As someone who grew up in an era of comparative bipartisanship and as a sitting U.S national security official who has served in two U.S. administrations, I am saddened that we have reached this point."
Support has been trending away from Rice for the past few days, according to a person familiar with the deliberations. That person spoke only on condition of anonymity, not authorized to discuss the situation publicly.
Attention now shifts to Kerry, who came close to winning the presidency in 2004 and has been seen as desiring the State job. In a statement, he made no mention of his own candidacy but praised Rice, who was an adviser to him his in his presidential bid.
Kerry was an early backer of Obama and was under consideration to become his first secretary of state. Obama has dispatched Kerry to foreign hot spots on his behalf. Kerry played the role of Republican Mitt Romney during Obama's presidential debate preparations this year.
The longtime senator would be almost certain to be easily confirmed by his colleagues on Capitol Hill.
Even if Rice had been chosen and confirmed, a contentious Senate fight could have sent her into the job with weakened support and used up some of the tough votes Obama may need from allies in the Senate later.
House Democratic women had cast the criticism of Rice as sexist and racist she is African-American and some expressed disappointment with the news.
"If judged fairly based solely on her qualifications for the job, she would've made an extraordinary secretary of state," said Rep. Karen Bass, D-Calif., a member of the House Foreign Affairs Committee.
Rice did not have a strong relationship with members of the Senate. Graham, who is the top Republican on the Appropriations subcommittee that handles foreign aid and the State Department, said he barely knew her.
In a brief statement, a spokesman for McCain said the senator "thanks Ambassador Rice for her service to the country and wishes her well. He will continue to seek all the facts surrounding the attack on our consulate in Benghazi."
Rice's decision comes ahead of the anticipated release next week of a report by an Accountability Review Board into the attack on the Benghazi mission. The report ordered by Clinton, focuses on the run-up to and the actual attack and is not expected to mention Rice's role in its aftermath.
Clinton is to testify about the report before Congress next Thursday.
At issue is the explanation Rice offered in a series of talk show appearances five days after the attack in Libya.
Rice has conceded in private meetings with lawmakers that her initial account that a spontaneous demonstration over an anti-Muslim video produced in the U.S. triggered the attack was wrong, but she has insisted she was not trying to mislead the American people. Information for her account was provided by intelligence officials.
Obama had been expected to announce his new national security team next week, but that could be pushed back because of fiscal cliff negotiations. The president may announce his nominees to lead the State and Defense Departments, and perhaps the Central Intelligence Agency, at the same time.
Hagel, a former Republican senator from Nebraska, is a Vietnam veteran, served two terms in the Senate and was a senior member of the Foreign Relations Committee. Obama and Hagel became close while they served in the Senate and traveled overseas together. Hagel has been critical of his party since leaving the Senate in 2008, saying the GOP had moved too far right.