"I just never saw me doing this all my life," Matheson said in an interview. "I always thought there would be other chapters in what I do in my public service career and this just seemed the right time to move on to the next opportunity."
Could that mean running for Senate or governor?
"Yes, sir," Matheson replied, quickly noting that he wasn't making any announcements now on that.
Matheson, son of former two-term Gov. Scott Matheson, had become Utah's most successful Democrat in decades and, because of that, the GOP's No. 1 target. Republicans poured millions into Utah trying to defeat the moderate, some say conservative, Democrat, coming at times only hundreds of votes short of ousting him.
His departure from the 4th Congressional District seat clears the way for an easier pickup for Republicans next year.
Catching his breath • State Sen. Jim Dabakis, the chairman of the Utah Democratic Party, spoke with Matheson before the announcement and said it was a sad day for Utah and America to lose someone in Congress who tries to get things done and break through the gridlock. After 14 years and with a young family, Matheson needed a break, Dabakis said.
"I understand it completely but the very things that Jim brings to the table, a willingness to roll up his sleeves and not be ideological and just go solve problems, that's the kind of thing we most need in Washington," Dabakis said.
Dabakis says after catching his breath and spending a few years at home, Matheson could be back.
"There's a lot of political life left in Jim Matheson," Dabakis said.
Saratoga Springs Mayor Mia Love who lost to Matheson by fewer than 800 votes last year has already started raising funds to challenge Matheson but said Tuesday his decision changes nothing.
"I certainly respect his decision and I wish him well in anything he decides to do," Love said. "We're going to keep going forward and raising money and getting our message out."
Matheson's exit helps that.
Past campaigns against the Democrat have attempted to tie him to liberals, like House Minority Leader Nancy Pelosi, and unpopular laws in Utah, like the Affordable Care Act. Love, or other Republicans who may jump into the race, now have an open seat to fight over while Democrats may not have a big enough name to mount a serious challenge, observers say.
Democratic dilemma • "It puts the seat in serious jeopardy for the Democrats, for sure," said Quin Monson, director of the Center for the Study of Elections and Democracy at Brigham Young University.
"The problem is the Democrats don't have a deep bench of people ready to move forward and they face a disadvantage in the partisan balance of the district, so I would assume that barring some major misstep on the part of the Republican, it increases the chances they'll take the seat," Monson said. "It also increases the chances someone else will jump into the race against Mia Love in the primary."
Monson said that Love had done a good job of keeping other Republicans out of the race Utah businessman Bob Fuehr was her only challenger so far and her fundraising advantage may still scare off challengers.
Months after a government shutdown and amid a deepening party divide on Capitol Hill, some long-time House members have opted to head back home rather than deal with the political climate. Republican Rep. Frank Wolf, a 34-year-veteran and dean of the Virginia delegation, announced Tuesday he wouldn't run for re-election, as did Rep. Tom Latham, R-Iowa.
Matheson says the partisan atmosphere may not be great but it's not what drove him out. He says last year would have been an easier time to call it quits faced with a new district and Mitt Romney heading the ticket but that he fought on and Utahns re-elected him. This time, he says, he spoke with his wife, Amy, and 15-year-old son, Will, and decided it was time.
"It was a great job," Matheson says. "It was an honor and I've met some amazing people and it's one of the most meaningful things I'll ever do in my life. But I think I want to take all that and apply it to someplace else now."
Matheson still has the remaining year of his term to serve.
Thad Hall, a political science professor at the University of Utah, said off-year elections those between presidential elections are tough times for incumbents like Matheson to run.
Higher office? • "I would assume this means Matheson has done polling that shows he can't win, so he's just going to cut his losses and go out on a high note, or he's going to run for governor or Senate and wants to save his war chest and time," Hall said.
Ditto, says Damon Cann, a political science professor at Utah State University, who said Matheson may have feared the long-term downside if he were to fall to Love next year.
"If he loses the election to Mia Love he can't run in the future for Senate or governor, but if he cuts his losses now, he's still a winner and he can run in the future for other offices," Cann said.
Utah Republican Party Chairman James Evans, though, says he's confident Matheson stands no shot at statewide office in Utah.
"I think it's much more difficult for a Democrat to win and if Jim Matheson chooses to run statewide, it would be my prediction that he couldn't prevail," Evans said. "Among the Democratic field, he would have the best possible chance, but I still don't believe he would be successful."
Evans did predict a bigger field now in the 4th District, adding, "I expect there will probably be more challengers."
That said, with Matheson out of the 4th District race, the state GOP can focus its resources elsewhere, like in state House races, Evans said.
Nationally, Republicans tried to play Matheson's announcement as bad news for all Democrats.
"This is a warning signal to Democrats coast to coast," said National Republican Congressional Committee Chairman Greg Walden, R-Ore. "Not only will this announcement allow Republicans to focus our energy and resources on defeating other vulnerable Democrats, but it also proves that Obamacare has become a total nightmare for any Democrat running in 2014."
Walden's counterpart, Democratic Congressional Campaign Committee Chairman Steve Israel, D-N.Y., praised Matheson for putting his constituents first.
"Jim's priorities have always been to focus on working together to solve our problems, responsibly put our fiscal house in order and make our country strong for the next generation," Israel said.
President Barack Obama praised Matheson for being "a forceful advocate of our nation's veterans and [working] to strengthen our economy through his support of key trade agreements. Michelle and I thank Congressman Matheson for his service and wish him, his wife Amy, and their two sons the very best in the future."
Robert Gehrke and Lee Davidson contributed to this report.