Center Ryan O'Reilly opened the scoring on a power play, assisted by Mitch Marner, who doubled the Canadians' lead shortly afterward.
But Mike Matheson was penalized for delaying the game in the third period and Herzog scored on the power play, then Vincent Praplan equalized.
Canada pushed hard for the winner only for Herzog to grab the overtime winner in Paris.
Earlier in Cologne, Russia routed Slovakia 6-0 and the United States fought back from two goals down to beat Latvia 5-3.
Russia reclaimed the lead of Group A and outshot Slovakia by 39 to 22.
Yevgeni Dadonov got the tournament favorites off to a great start with just 1:12 played, then claimed his second goal on a power play. Andrei Mironov made it 3-0 at the first interval.
Nikita Kucherov and Ivan Telegin added goals in the second period, and Vladislav Gavrikov finished the scoring as Russia stayed two points above the U.S.
It started badly for the U.S. with Latvia's Teodors Blugers opening the scoring midway through the first period and Kaspars Daugavins making it 2-0 at the start of the second, assisted by Andris Dzerins.
Anders Lee and Danny Dekeyser were penalized for tripping, and Dzerins for holding, as tempers frayed.
Lee came back to set up J.T. Compher's goal for the U.S., but another moment of indiscipline from Clayton Keller gave the Latvians yet another power play chance, duly taken by Oskars Cibulskis on a rebound after Connor Hellebuyck blocked Miks Indrasis' initial shot.
Nick Bjugstad pulled one back again and Johnny Gaudreau equalized for the U.S. before the end of the second period with his fourth of the tournament.
Andrew Copp's and Dylan Larkin's goals then ensured the U.S. stretched its winning run to four games, following its surprise defeat to co-host Germany.
"It was a huge win given the early adversity," Copp said. "We needed to get our legs under us a little bit and get into intensity mode."
Germany boosted its quarterfinal hopes with a 4-1 win over Italy thanks to goals from Christian Ehrhoff, Matthias Plachta, Yannic Seidenberg and Dominik Kahun. Michele Marchetti got Italy's goal after Leon Draisaitl set up Ehrhoff's opener in his first start. Draisaitl has just arrived from the NHL playoffs.
Germany moved level with fourth-placed Latvia in Group A. The teams meet for what will likely be a decisive game on Tuesday. The top four in each group go through.
Belarus defeated Slovenia 5-2 for its first win of the tournament in Paris, where Markus Hannikainen scored the winner for Finland to beat Norway 3-2 to go fourth in Group B.