Bergdahl, held captive for almost five years, was released May 31 in exchange for five Taliban prisoners held by the U.S. at the Guantanamo Bay prison camp in Cuba. The deal requires that the five men remain in Qatar for at least a year.
"They're not the only ones keeping an eye on them," Kerry said of the Qataris, while declining to elaborate.
"We have proven what we are capable of doing with al-Qaida," he said.
The threats by the Taliban prisoners to resume fighting in Afghanistan and kill Americans amount to propaganda, Kerry said. "They'll say whatever they want to stir the waters," he said.
With the U.S. combat role in Afghanistan coming to an end and most troops to be withdrawn by the end of 2016, opportunities for Taliban to kidnap Americans will dwindle, Kerry said.
Whether or not Bergdahl deserted his post in eastern Afghanistan, as some of his former fellow soldiers claim, "it would have been offensive and incomprehensible to consciously leave an American behind," Kerry said.
Sen. John McCain, an Arizona Republican and former Vietnam prisoner of war, criticized the deal on the same program.
The five Taliban prisoners "were evaluated and judged as too great a risk to release," McCain said.
"I'm sad to tell you I'm afraid they're going to re-enter the fight," he said.
Lawmakers of both parties have faulted President Barack Obama's administration for failing to notify and consult them about the prisoner exchange in advance.
Sen. Dianne Feinstein, the California Democrat who heads the Senate intelligence committee, echoed that concern Sunday as she questioned Kerry's assertion that the five freed Taliban members would be carefully monitored.
"It's hard to be comfortable when you really haven't been briefed on the intricacies of carrying out this agreement," Feinstein said on CBS' "Face the Nation" program.
Bergdahl, who remains at an Army medical center in Landstuhl, Germany, has told medical officials that his former captors locked him in a metal cage in darkness for weeks at a time as punishment for trying to escape, the New York Times reported Sunday. A U.S. defense official declined to confirm that account.
Feinstein, when asked about the Times report, said she had heard rumors of Bergdahl trying to escape his captivity, though she said she had never heard of any allegations of torture.
Breaking with the administration over its handling of the prisoner swap, Feinstein said she wasn't reassured by Kerry's claim that the five Taliban men would be monitored in Doha.
"You can't help but worry about them in Doha," Feinstein said. "And we have no information on how the United States is actually going to see that they remain in Doha, that they make no comments, that they do no agitation."
Republican lawmakers said the administration violated a law requiring 30 days' notice to Congress before any prisoners are transferred or released from Guantanamo Bay. Defense Secretary Chuck Hagel and National Security Advisor Susan Rice said a week ago they couldn't afford to wait 30 days because Bergdahl's health was deteriorating and his life was at risk.
Sen. Saxby Chambliss of Georgia, the top Republican on Feinstein's committee, disputed that explanation today on CBS.
"No intelligence supported that," Chambliss said. "And now they come back and because he is in decent health, considering where he's been, they've changed their story. They said, well, you know, we suspect his life may be in danger, if word got out of this pending possible trade that his life may be in danger. Again, I can just tell you there is no intelligence to support that."
In signing the notification requirement into law as part of last year's defense policy bill, Obama issued an accompanying statement asserting his right as commander in chief to make exceptions.
Former Attorney General Michael Mukasey, who served in the George W. Bush administration, said Obama "broke the law" by failing to notify Congress of the prisoner swap. "But I believe that the law itself is unconstitutional" because it restricts Obama's ability to conduct foreign policy as commander in chief, Mukasey said on "Fox News Sunday."
Rep. Mike Rogers of Michigan, the Republican chairman of the House intelligence committee, said the five Taliban men were allowed too much freedom to operate in Qatar.
"They can meet with Taliban political leaders in Qatar," he said. "They can have family members travel to Qatar and back to Pakistan and Afghanistan, and we believe that's certainly an opportunity for a courier network to get them prepared for what's next."
The prisoner swap sets a bad precedent that has become "a huge regional and geopolitical problem for the United States," Rogers said. "Hostages are now currency in this war on terror. That's always dangerous for both diplomats, aid workers, soldiers on the battlefield."
Instead of a prisoner swap, Rogers said, the U.S. should have pressured Pakistan to help release Bergdahl because the Army soldier "was in Pakistan" while being held captive.
The Pentagon has yet to explain the circumstances of Bergdahl's capture from a remote combat outpost in eastern Afghanistan. Several soldiers who served in his unit have gone public in recent days with claims that he deserted them by walking away from their base. They also have said some of their fellow soldiers were killed in missions to search for Bergdahl.
"Every single mission after Bergdahl left was tilted towards finding Bowe Bergdahl," said Joshua Cornelison, a medic who served in Bergdahl's unit.
"The American public needs to be educated that Bowe Bergdahl is not a hero, that he did not serve with honor or any kind of distinction, that he is a deserter that purposely and willfully walked away from his post and he needs to be held 100 percent accountable" for putting soldiers' lives in danger, Cornelison said on "Fox News Sunday."