The band members said their protest was meant to raise their concern about increasingly close ties between the state and the church.
Russian parliament passed the amnesty bill last week, allowing the release of thousands of inmates. Alekhina and Tolokonnikova, who were due for release in March, qualified for amnesty because they have small children.
There has been an international outcry over Russia's human rights record, including for passing a law earlier this year that bans so-called homosexual propaganda among minors, which gay groups in Russia and abroad say feeds the existing enmity toward gay people in the country.
Tolokonnikova walked out of a prison in the eastern Siberian city of Krasnoyarsk on Monday, smiling to reporters and flashing a V sign.
"How do you like our Siberian weather here?" said Tolokonnikova, wearing a down jacket but no hat or scarf in -25 degrees Celsius (-13 degrees Fahrenheit). Tolokonnikova said that she and Alekhina will set up a human rights group to help prisoners.
Tolokonnikova said the way prisons are run reflect the way the country is governed.
"I saw this small totalitarian machine from the inside," the 24-year-old said. "Russia functions the same way the prison colony does," she said.
Alekhina, who was released earlier on Monday from a prison outside the Volga river city of Nizhny Novgorod, said she would have stayed behind bars to serve her term if she was free to turn it down.
"If I had a chance to turn it down, I would have done it, no doubt about that," she told Dozhd TV. "This is not an amnesty. This is a hoax and a PR move."
She said the amnesty bill covers less than 10 percent of the prison population and only a fraction of women with children behind bars. Women convicted of grave crimes, even if they have children, are not eligible for amnesty.
Alkhina said that prison officials did not give her a chance to say goodbye to cell mates, but put her in a car and drove her to the train station in downtown Nizhny Novgorod. Before seeing her family and friends, she met with local rights activists and said she will work on defending human rights.
The release of the two Pussy Riot band members came days after President Vladimir Putin pardoned Mikhail Khodorkovsky, a former oil tycoon and once Russia's richest man, who spent a decade in prison after challenging Putin's power. Khodorkovsky flew to Germany after release and said he will stay out of politics. He pledged, however, to fight for the release of political prisoners in Russia.
Russia's Supreme Court earlier this month ordered a review of the Pussy Riot case, saying that a lower court did not fully prove their guilt and did not take their family circumstances into consideration when reaching the verdict.
Also on Monday, the European Court of Human Rights said it will review a complaint filed by band members over their treatment while on trial in Moscow in 2012.
Vasilyeva reported from Moscow.