Brunet confirmed two more deaths early Sunday afternoon after confirming two people were found dead overnight. One death was confirmed Saturday.
All but one of the 73 cars were filled with oil, which was being transported from North Dakota's Bakken oil region to a refinery in Saint John, New Brunswick.
The eruptions early Saturday morning sent residents of Lac-Megantic scrambling through the streets under the intense heat of towering fireballs and a red glow that illuminated the night sky.
Local Fire Chief Denis Lauzon likened the charred scene to "a war zone."
"This is really terrible. Our community is grieving and it is taking its toll on us," Mayor Colette Roy-Laroche said.
On Sunday afternoon, Prime Minister Stephen Harper toured the town where a large part of the downtown area has been leveled.
"This is an unbelievable disaster," Harper said. "This is a very big disaster in human terms as the extent of this becomes increasingly obvious."
Harper said the whole country is worried about the missing and is praying for the town.
"This is an enormous area, 30 buildings just completely destroyed, for all intents and purposes incinerated," Harper said. "There isn't a family that is not affected by this."
The search for victims in the charred debris was hampered because two tanker cars were still burning Sunday morning, sparking fears of more potentially fatal blasts.
Lauzon said firefighters are staying 500 feet (150 meters) from the burning tankers, which are being doused with water and foam to keep them from overheating.
The multiple blasts came over a span of several hours in the town of 6,000, which is about 155 miles (250 kilometers) east of Montreal and about 10 miles (16 kilometers) west of the Maine border. It is a picturesque lakeside town in Quebec's Eastern Townships.
The derailment caused at least five tanker cars to explode in the downtown district, a popular area packed with bars that often bustles on summer weekend nights. Police said the first explosion tore through the town shortly after 1 a.m. local time. The fire then spread to several homes.
Brunet said he couldn't say where the bodies were found exactly because the families have not been notified. Many feared for the lives of those who were at the Musi-Cafe bar where dozens of people were enjoying themselves in the wee hours of a glorious summer night.
Residents who gathered outside a community shelter Sunday hugged and wiped tears as they braced for bad news about missing loved ones.
Henri-Paul Audette headed there with hope of reuniting with his missing brother. Audette, 69, said his brother's apartment was next to the railroad tracks, very close to the spot where the train derailed.
"I haven't heard from him since the accident," he said. "I had thought ... that I would see him."
Another man who came to the shelter said it's difficult to explain the impact this incident has had on life in Lac-Megantic. About a third of the community was forced out of their homes. David Vachon said he has one friend whose sister is missing and another who is still searching for his mother.
The cause of the accident was believed to be a runaway train, the railroads operator said.
Edward Burkhardt, the president and CEO of Rail World Inc., the parent company of Montreal, Maine & Atlantic Railway, said the train had been parked uphill of Lac-Megantic because the engineer had finished his run. The tanker cars somehow came loose and sped downhill nearly seven miles into the town before derailing.
"We've had a very good safety record for these 10 years," Burkhardt said of the decade-old railroad. "Well, I think we've blown it here."
Joe McGonigle, Montreal, Maine & Atlantic's vice president of marketing, said the company believes the brakes were the cause. He said the rail company has been in touch with Canada's Transportation Safety Board.
"Somehow those brakes were released and that's what is going to be investigated," McGonigle said in a telephone interview Sunday. "We're pretty comfortable saying it is the brakes. The train was parked, it was tied up. The brakes were secured. Somehow it got loose."
Lauzon, the fire chief, said that firefighters in a nearby community were called to a locomotive blaze on the same train a few hours before the derailment. Lauzon said he could not provide additional details about that fire since it was in another jurisdiction. Nantes Fire Chief Patrick Lambert couldn't be immediately reached, but McGonigle confirmed the fire department showed up after the first engineer tied up and went to a local hotel and after someone reported a fire.
"We know that one of our employees from our engineering department showed up at the same time to assist the fire department. Exactly what they did is being investigated so the engineer wasn't the last man to touch that train, we know that, but we're not sure what happened," McGonigle said.
McGonigle said there was no reason to suspect any criminal or terror-related activity.
Because of limited pipeline capacity in North Dakota's Bakken region and in Canada, oil producers are increasingly using railroads to transport much of the oil to refineries on the East, Gulf and West coasts, as well as inland. Harper has called railroad transit "far more environmentally challenging" while trying to persuade the Obama administration to approve the controversial Keystone XL pipeline from Canada to the Gulf Coast.
The proliferation of oil trains has raised concerns of a major derailment like this. McGonigle said it is a safe way to transport oil.
"There's much more hazardous material that moves by rail than crude oil. We think it is safe. We think we have a safe operation. No matter what mode of transportation you are going to have incidents. That's been proven," McGonigle said. "This is an unfortunate incident."
Myrian Marotte, a spokeswoman for the Canadian Red Cross in Lac-Megantic, said there are about 2,000 evacuees and said 163 stayed at their operations center overnight.
Patrons gathered at a nearby bar were sent running for their lives after the thunderous crash and wall of fire blazed through the early morning sky early Saturday. Bernard Theberge, who was outside on the bar's patio at the time of the crash, feared for the safety of those inside the popular Musi-Cafe when the first explosion went off.
"People started running and the fire ignited almost instantaneously," he said.
"It was like a movie," said Theberge, who considered himself fortunate to escape with only second-degree burns on his right arm. "Explosions as if it were scripted but this was live."
According to Montreal Maine & Atlantic's website, the company owns more than 500 miles (800 kilometers) of track serving Maine, Vermont, Quebec and New Brunswick.
Montreal, Maine and Atlantic carried nearly 3 million barrels of oil across Maine last year. Each tank car holds some 30,000 gallons (113,600 liters) of oil.
Maine state officials were notified regarding concerns about the smoke from the fire but staff meteorologists don't believe it will have a significant impact, Peter Blanchard of the state Department of Environmental Protection said Sunday.
The Maine environmental agency had previously begun developing protection plans for areas in the state through which the oil trains travel.
But Glen Brand, director of the environmentalist Sierra Club's Maine chapter, said the Quebec derailment is reason enough to call for an immediate moratorium on the rail transport of oil through the state.
"This tragic accident is part of the larger problem of moving oil through Maine and northern New England," Brand said. "It reinforces the importance of moving away from dirty fossil fuels that expose the people of northern New England, Maine and Quebec to a host of dangerous risks."
French President Francois Hollande's office issued a statement offering condolences to the victims in the predominantly French-speaking Canadian province.
Associated Press writer Rob Gillies and Charmaine Noronha contributed from Toronto.