He argues Pelosi has contributed to the polarization in Washington that has squeezed out moderates in each party and made it more difficult for Congress to take action on pressing issues.
"If we had new leadership, that helps create a new opportunity for working in a constructive way," Matheson said.
Matheson has no intention of running as an alternative to Pelosi. She has led the Democrats since 2003 and remains popular with the party's liberal base. She's also the first woman to head a congressional party and the first to be House Speaker
"I would not consider putting my name up there," Matheson said.
Democrats gained eight House seats in the recent election, but need another 17 to recapture control of the House.
Matheson had cast a vote in favor of Pelosi as party leader three times, but that ended after the 2010 Republican landslide, where the GOP picked up 63 seats and control of the House.
During that campaign, Matheson faced hundreds of yard signs that read Matheson=Pelosi and after the election he made it clear that those signs were not accurate.
He slammed Pelosi for not reaching out to moderate and conservative Democrats and for pushing an agenda that didn't focus more heavily on jobs and the economy. Matheson nominated his friend, Health Shuler, a fellow conservative Democrat who represented North Carolina, to be the party's leader.
Shuler, who did not run for reelection in 2012, ended up getting just 10 votes and in all only 18 Democrats opposed Pelosi.
While Matheson campaigned against Pelosi, Matheson also became part of the broader Democratic leadership team. He accepted a role as a chief deputy whip for House Majority Whip Steny Hoyer, who has been Matheson's congressional mentor.
As part of the nine-member whip team, Matheson has kept tabs on the votes of Blue Dogs, a coalition of House moderates, which has taken serious hits in recent years dropping from a high of more than 50 seats in 2006, to just 15 seats after the most recent election.
He expects to stay in that role going forward, though the leadership votes won't take place until after the Thanksgiving holiday.