The tumult stems from a written confession that Fife submitted to residents in early December, telling them of his extramarital affair with a woman he had formerly counseled while serving as a Mormon bishop.
Fife, 67, asked for their forgiveness and support, remaining resolute about continuing as the city's mayor.
However, several residents voiced strong sentiment Thursday that Fife a husband, father and grandfather should step down and let someone else steer the city.
Suresh Kulkarni, who owns property in Brigham City and chairs the community hospital's board of trustees, said he was representing many who have called him, emailed him or talked to him about Fife's situation, urging his resignation.
"As long as Mayor Fife remains in that chair, he is going to be a lightning rod for continued bad-mouthing of the city," Kulkarni said. 'We have no idea how many more bombshells, lawsuits or whatever else might be coming at us."
Most of the audience stood when Kulkarni asked who favored stripping the mayor of his powers.
Brigham City resident Sherry Phipps said that extramarital affairs shatter trust, and that the road to regaining trust in a family situation can be long and difficult.
"Many people who commit sexual transgressions fail to realize the impact it has on others," Phipps said, adding that the mayor's affair had put a cloud over the city.
Brigham City resident Mike Busby blasted the mayor for committing adultery.
"It was of his own free will and accord that he did what he did," Busby said, adding that he is sad for Fife but will not apologize on his behalf.
No one spoke at this meeting in support of Fife continuing as Mayor.
Following the vote, Kulkarni said the council's action fell short.
"The problem with this is you still have him sitting there and he continues to bring ridicule upon the city," Kulkarni said. "If I was him, I would have just walked out and said 'I'm done.'"
Kris Powers, a partner in the Leavitt Group of Northern Utah, said she was grateful the ordinance passed but said it wasn't enough.
"I really think we need our mayor to step down or we need to find an avenue to force him out," Powers said, saying she would support a state recall law for matters that are "really serious."
Rep. Lee Perry, R-Perry is drafting such a bill and hopes to garner support for it from fellow lawmakers during the Legislative session that starts in January.