Vande Velde tried to outsprint Fedrigo in the last 200 yards, but his rival accelerated to the line to become the fourth French rider to win a stage this Tour.
"It's incredible. ... The stars need to be aligned," said Fedrigo, whose last Tour stage victory was also in Pau two years ago. "There are days when things go like that."
Fedrigo said the victory was especially satisfying because he was sidelined for six months last year with Lyme disease.
Vande Velde said he had little hope of winning the sprint because of his inexperience in such finishes.
"I knew my chances were not good at all with Fedrigo. He is really quick guy and has got a much better sprint," said Vande Velde, who wore the Giro d'Italia leader's pink jersey for a day in 2008 and was part of the Garmin-Cervelo squad that won the Tour team trial last year.
"I've never had a sprint [victory] ever," said Vande Velde, who is 36. "I've always been a slow-twitch guy. But I'm still happy with my ride. I wish there was a hill to finish it up on, but that's the way it goes."
Wiggins finished 11 minutes, 50 seconds behind Fedrigo in the main pack. The 32-year-old Team Sky leader is looking to become Britain's first winner of cycling's showcase race
Overall, Wiggins leads second-place teammate Christopher Froome by 2:05. Vincenzo Nibali of Italy is third, 2:23 behind. Defending champion Cadel Evans remains fourth, 3:19 back.
Wiggins said he and other riders were looking forward to Tuesday's rest day.
"A lot of people are tired now mentally more than physically," he told French TV. "It was hard today nonetheless, it was quite hot. And the road was hilly."
Tour organizers said another six riders dropped out of the race, including France's Sylvain Chavanel because of illness. Brett Lancaster of Australia withdrew because of back pain from crashes in Saturday's 13th stage, his Orica-GreenEdge team said. Forty-two riders have now quit the race, trimming the pack to 156.
Tour organizers completed their tally from Sunday when tacks were thrown on the course. Race director Jean-Francois Pescheux said the 30 top riders had total of 61 flat tires because of the small nails. Also, hundreds of cars, bikes and motorcycles in the race convoy were affected. He said police are investigating.
The rest day comes before two punishing days in the Pyrenees. Those rides, along with a time trial in the next-to-last stage, are likely to determine the winner of the three-week race, which ends July 22 in Paris.
Tour de France
P Wednesday, 4:30 a.m., TV • NBCSN