BYU never trailed in overtime, taking a quick 66-64 lead on Kyle Collinsworth's rebound basket, and it held on by making just enough free throws in the last few minutes.
The Cougars (23-10) will play for the WCC tournament championship at 7 p.m. MDT Tuesday against Gonzaga. San Francisco fell to 21-11.
Tyler Haws scored 24 points despite an off shooting night (5 of 19), and Collinsworth added 18 points and 12 rebounds in the win. Haws went 14 of 14 from the free-throw line, and the Cougars combined to go 24 of 28 from the stripe.
Anson Winder had 15 points, including a free throw with 8.4 seconds left that provided the final margin. Collinsworth, a 57 percent free-throw shooter, made a pair with 24 seconds left in OT to give the Cougars a 76-73 lead.
"My first thought was just my teammates," Collinsworth said. "I knew I had to make them. I knew I was going to make them."
It was the first WCC semifinal to go to overtime since 2006.
San Francisco got off a last shot to win it, but Avry Holmes' 3-point attempt was off. The Cougars beat the Dons for the third straight time this season.
"It just feels good to win as a team," Collinsworth said.
Trailing at halftime, the Cougars regained what once was a 10-point lead when Haws followed his own miss, but the advantage swung back and forth from there until BYU put together a little spurt and moved ahead by seven, 58-51, with 10:10 remaining.
Four straight points by Haws with about six minutes left gave BYU an eight-point lead, but the Dons wouldn't go away. They reeled off an 8-0 run, tying it on a Mark Tollefsen 3-pointer from the corner with 3:20 left.
Neither team scored in regulation after that.
"I thought we gave a winning effort. Sometimes you come up a little bit short," USF coach Rex Walters said.
Collinsworth said the Cougars paid no attention to Walters' pre-game boast that he had the best team in the tournament this year. Asked about that after the game, Walters stuck to his guns, saying he wouldn't trade his team for anything.
"I would love another shot at BYU," he said. "They won the game tonight. They deserved it."
The Cougars went into the locker room trailing 37-35 despite leading by 10 points several times in the first half.
Rose changed his starting lineup, going with freshman Eric Mika instead of Luke Worthington. Mika delivered early, not picking up his first foul until 13:23 remained in the first half and scoring eight points in the first eight minutes.
The Cougars started fast and led 22-12 when Haws made a pair of free throws with 11:58 remaining in the first half. But the 65 percent shooting wouldn't last.
BYU led 29-20 on two more Haws free throws with 7:24 left in the half, but San Francisco went on an 11-0 run to take its first lead. Collinsworth ended the Cougars' long scoring drought after five straight empty possessions, but by then USF was back in a game BYU controlled for the first 15 minutes.
USF's Holmes hit a 3-pointer with 38 seconds left in the half to give the Dons the two-point lead.
Dickerson had 12 points in the first half for USF, while Haws had 10 for BYU. Dickerson finished with 20 points.
"Just war, that's basically all it was," Dickerson said about the third meeting between the teams this season.
BYU 79, San Francisco 77 in OT
• Tyler Haws scores 24 points despite 5 of 19 shooting, and the Cougars advance to the WCC championship game with a razor-close overtime win
• Haws goes 14 for 14 and the Cougars go 24 of 28 from the free-throw line in the semifinal game
• Kyle Collinsworth scores 18 points and adds 12 rebounds for BYU