The Spartans improved to 9-2 with the mild upset, while the Cougars fell to 6-5.
Nelson completed 28 of 51 passes for 335 yards and a touchdown, with one costly interception
San Jose State quarterback David Fales was 24-of-34 for 305 yards and three TDs.
The rain stayed away until the fourth quarter, while the Cougars' No. 4-ranked defense was tardy for the first 30 minutes. The one that showed up first was befuddled by SJSU's offense as the Spartans converted their first eight third-down opportunities in the first half and jumped out to a 20-7 lead at the break.
BYU's offense finally found its rhythm in the fourth quarter, as Nelson threw a 20-yard touchdown pass to David Foote on fourth-and-10 to cut the deficit to 20-14 with 2:36 remaining.
Hoffman then jumped on an onside kick, giving the Cougars life.
Trailing 20-7, the Cougars drove to the SJSU 25 early in the fourth quarter, but Nelson was intercepted by Jimmy Pruitt near the goal line.
The Cougars forced a punt and again drove deep into SJSU territory. But BYU couldn't score on four tries inside the 7-yard line, turning the ball back to San Jose with five minutes, 10 seconds remaining. Those red-zone failures will haunt the Cougars the rest of the season.
San Jose State dominated the first half on both sides of the ball.
The Spartans scored touchdowns on their first three possessions, driving 65 yards on their first series, 75 on their second and 71 on their third.
It could have been worse for the Cougars, but safety Daniel Sorensen intercepted a pass to stop SJSU's fourth drive.
After San Jose State's first touchdown, the Cougars drove 79 yards and capped the drive with Jamaal Williams' 16-yard touchdown run to take a 7-6 lead.
That was short-lived, however, as the Spartans answered with a nine-play, 75-yard drive to regain the lead they would not relinquish.
Fales was 18-for-23 for 237 yards and three touchdowns in the first half alone and became the school's single-season record-holder for TD passes.
Hoffman caught three passes for 53 yards in the first half and has now caught at least one pass in 30 straight games. He finished with 13 catches for 155 yards.
The Cougars finished with 422 total yards, while San Jose State had 364, including 295 in the first half.
Storylines SJSUis explosive
R In short • Fired-up San Jose State uses an offensive explosion in the first half and holds on to defeat BYU and exact revenge for last year's loss in Provo.
Key moment • San Jose State drove 65, 75 and 71 yards on its first three possessions to take a 20-7 halftime lead.