The NSA has argued that collecting metadata some of which is likely to be stored at the NSA's Utah Data Center is a powerful instrument in being able to determine if terrorists are communicating with people inside the United States.
Alexander, testifying before the panel for the first time since September, said he is open to suggestions on a better way to achieve the same goal but said neither the agency nor private industry has a better idea.
"If we let this down, I think we'll have let the nation down," Alexander said.
Senate Judiciary Chairman Patrick Leahy, D-Vt., started off Wednesday's hearing noting recent press reports that the NSA collects some 5 billion phone records a day and has reportedly gained access to monitor the online-gaming world.
"It's clear we have more oversight to do," Leahy said, adding later that, "just because you can do something does it make sense to do it?"
Wednesday's oversight hearing is one of a series prompted by former NSA contractor Edward Snowden's leaks of secret court orders and agency documents that show a much broader, and more intense, level of spying from the government previously acknowledged.
President Barack Obama said in an interview with MSNBC's Hardball last week that he plans to propose some self-restraint initiatives on the NSA "that can give people more confidence" in government actions.
"The people at the NSA, generally, are looking out for the safety of the American people," Obama said. "They are not interested in reading your emails. They're not interested in reading your text messages. And that's not something that's done. And we've got a big system of checks and balances, including the courts and Congress, who have the capacity to prevent that from happening."
Alexander said he's feeling the brunt of more oversight since the Snowden leaks and other administration officials say they are hopeful that they can explain the NSA's efforts to help people become more comfortable. But Robert Litt, the general counsel of the Office of the Director of National Intelligence, said there should be a careful balance in bringing more eyes on the NSA.
"If there are ways we can do that better, we're open to that," Litt said. "We do think that it's important in considering [changes] that we don't throw the baby of national security out with the bath water of oversight."