The two-term congressman stood before his Republican colleagues earlier this year and sarcastically promised never to sign a pledge again. He was voicing his frustration with advocacy groups such as the National Taxpayers Union that supported the final debt-limit compromise even though it fell far short of the Cut, Cap and Balance pledge that it had backed for months.
"I was bemoaning those who abandoned their own pledges," he said.
Chaffetz insists he doesn't regret any of the pledges he signed nor does he have any intention of backing away from them, but he remains concerned about the power they give outside groups, especially with a major debate about tax reform in the offing.
He isn't the only one.
A number of conservatives are concerned about the power Grover Norquist and his Americans for Tax Reform wield over Republicans through their "taxpayer-protection pledge" signed by all but six House Republicans and seven Senate Republicans.
The four GOP members of Utah's delegation all signed Norquist's pledge, promising to oppose any tax increases and pair the elimination of a tax deduction or credit with a tax cut elsewhere.
Earlier this month, Rep. Frank Wolf, R-Va., one of the few House Republicans not to sign, went to the House floor to criticize Norquist's influence.
"I believe how the 'pledge' is interpreted and enforced by Mister Norquist," he said, "is a roadblock to realistically reforming our tax code."
The next day Sen. John Thune, R-S.C., went on MSNBC and said, "We shouldn't be bound by something that could be interpreted different ways if what we're trying to accomplish is broad-based tax reform."
Chaffetz said he isn't trying to take on Norquist's pledge directly, but worries how some could construe votes in Congress.
"You have to look at each situation as they come up," he said. "If we are going to get rid of the tax loophole for rum manufacturers, do we need a corresponding decrease in taxes somewhere else? I think that is a legitimate question."
Norquist doesn't see that as a tough call.
"If you want to keep your commitment to the voters of Utah," he said, "just make sure there is an offsetting tax cut."
Norquist doesn't think the taxpayer-protection pledge, first offered in 1986, is confusing or open to interpretation in the slightest.
"There is never a question on whether it was a tax increase or not," he said. "The only time people say, 'Oh, that is up to interpretation' is when they are trying to raise taxes."
He said the value of his pledge is that it locks candidates into a position and has made it uncomfortable for politicians who go back on their word. He referenced former President George H. W. Bush, who famously said "read my lips, no new taxes" but then signed a tax increase.
All of the current GOP presidential candidates have signed Norquist's pledge save for one.
Jon Huntsman refuses to sign a single political pledge, a position he has stuck to since his time as Utah's governor.
"I have a pledge to my wife and I pledge allegiance to my country, but beyond that no pledges," Huntsman said during a presidential debate. "I think it diminishes the political discussion. I think it jeopardizes your ability to lead once you get there."
Rep. Rob Bishop, R-Utah, said he is generally "somewhat skeptical" of pledges. He was the only Republican in the state's delegation who didn't sign the Cut, Cap and Balance pledge earlier this year, because he wanted more flexibility in seeking a compromise. He ultimately voted against the package put together by party leaders.
Bishop doesn't see Norquist's anti-tax pledge as an impediment to reforming the tax code. He said it is an impediment to a tax increase.
Norquist said his pledge has also helped build the Republican brand, giving the voters some assurances when they enter the polling station. And that's something politicians with pledge fatigue should think about.
"If you vote for the Republicans," he said, "they may invade a small country with a name they can't pronounce, but they will not raise your taxes."