The Finance Committee hearing was the second on the topic and the first to include former IRS Commissioner Doug Shulman, who was in charge for the 18 months when IRS agents targeted groups with "tea party," "patriot" and "9/12" in their name, or those that criticized the government or proposed a smaller government.
Shulman declined to directly apologize when repeatedly asked to do so by Sen. John Cornyn, R-Texas.
"I certainly am not personally responsible for creating a list that had inappropriate criteria on it," Shulman said. "I very much regret that it happened and it happened on my watch."
Democrats and Republicans on the committee and throughout Congress expressed outraged at the findings of the inspector general report, with Finance Committee Chairman Max Baucus, D-Mont., saying: "The IRS abandoned good judgment and lost the public's trust."
Hatch, the panel's top Republican, said it went beyond that.
"There appears to have been more than a hint of political bias in the IRS's processing of applications of groups applying for tax-exempt status," he said.
Hatch zeroed in on Miller, who he said misled him by offering technical responses to his 2012 letters at a time when he knew that the targeting took place.
"You just sat on that guilty knowledge," he said. "That is a lie by omission."
"I did not lie, sir," Miller responded.
Hatch then said: "You knew this was going on. You knew we were concerned. Why didn't you let us know?"
"I believe I did answer them," Miller said, "and answer them truthfully, sir."
Shulman testified in the House in early 2012 that such targeting wasn't taking place, though he found out a few months later that he was incorrect. He said he only had partial information on the targeting and decided it made more sense to let the audit continue rather than immediately inform Congress of the activity.
Miller also pushed back against the accusation that the inappropriate targeting, which included requests for disclosure of donors and political beliefs, was motivated by politics.
"We were not politically motivated in targeting conservative groups," he said, instead apologizing for bad management. "I think what happened here was foolish mistakes were made by people trying to be more efficient."
Sen. Mike Crapo, R-Idaho, said Miller's argument that agents acted with no political motivation is "almost beyond belief" and he questioned inspector general J. Russell George on the topic.
George said he found no political motivation and based that on interviews of employees involved though the interviews were not under oath. The committee is expected to probe that matter further.
Senators also wanted to know why IRS leaders didn't take stronger action when they first learned about the targeting.
Miller said an employee who sent inappropriate letters to tea party groups was reassigned and another employee received "oral counseling," a response that senators found lacking.
Members of both parties sought to broaden the probe.
Republicans wanted to know how the IRS mistakenly released confidential applications of groups to the reporting website ProPublica. Those disclosures included information on the group Freedom Path, which spent at least $570,000 supporting Hatch's re-election in 2012.
Democrats criticized IRS officials for failing to adequately oversee the rise in politically active tax-exempt groups, which are used in part to keep the identities of its donors private, and argued Congress should revamp tax laws governing these groups.
The House Oversight and Government Reform Committee, which includes Rep. Jason Chaffetz, R-Utah, will start its probe of the IRS's action in a hearing on Wednesday.