The White House and IRS leaders initially laid the blame on two mid-level employees, including one, Elizabeth Hofacre, who asked to be transferred off of the tea party cases back in 2010.
Chaffetz questioned Hofacre at the hearing, asking her if it was accurate that "rogue IRS agents" did this on their own.
"I was following directions from management and they were aware of what I was doing," she said. "I know the accuracy of any inference to rogue agents is not correct."
Chaffetz then launched into a fiery critique of the White House.
"When you have the spokes-person of the president of the United States make a definitive statement that it was two rogue agents and start poking at these people, who have no power to do anything about it, that is wrong," he said. "How dare anybody suggest we are at the end of this. This is the beginning of this. We have to make an example of it. We need to get to the bottom of it."
He also criticized Cummings, saying if the committee followed his lead the investigation would have already ended.
Cummings took issue with Chaffetz's comments saying he wants the investigation to continue but wants Republicans to stop inferring White House involvement. But Chaffetz argued the White House is involved.
"You have the White House spokesman, Jay Carney, blaming one of the people sitting there at this table," he said, referring to Hofacre. "You have the former acting commissioner saying it was two rogue agents. It was the most powerful people in Washington, D.C., placing blame on a person sitting before us and that's why it continues forward and that's why I do think the White House is now engaged in this."
Washington IRS employees were aware of and in some cases involved in the tea party cases and new documents show that some progressive groups also received inappropriate scrutiny.
The House Oversight committee is continuing to interview IRS agents who were involved. A separate bipartisan investigation is underway in the Senate Finance Committee and is jointly led by Sens. Max Baucus, D-Mont., and Orrin Hatch, R-Utah.