"Inordinate and unpredictable delay has resulted in a death penalty system in which very few of the hundreds of individuals sentenced to death have been, or even will be, executed by the State," wrote Carney, a George W. Bush appointee.
He noted that death penalty appeals can last decades and as a result most condemned inmates are likely to die of natural causes before their executions are carried out.
Carney also wrote that since the current death penalty system was adopted by California voters 35 years ago, more than 900 people have been sentenced to death, but only 13 have been executed.
"As for the random few for whom execution does become a reality, they will have languished for so long on Death Row that their execution will serve no retributive or deterrent purpose and will be arbitrary," the judge stated.
Gil Garcetti, a former Los Angeles County district attorney, who has become an anti-death penalty activist called the ruling "truly historic."
"It further proves that the death penalty is broken beyond repair," he said, calling for capital punishment to be replaced with "with life in prison without the possibility of parole."
Carney's ruling came in a legal petition brought by Ernest Dewayne Jones, sentenced to die in 1994 after being convicted of murdering and raping his girlfriend's mother.
Jones remains on death row "with complete uncertainty as to when, or even whether, it will ever come," the judge wrote, adding, "Mr. Jones is not alone."
Carney's ruling could be appealed by the governor or state attorney general, who both oppose the death penalty. For now, Jones will likely remain on death row.
Carney noted that "arbitrary factors" such as the manner in which paperwork is handled are what "determine whether an individual will actually be executed."
Another federal judge put California's death penalty on hold in 2006 when he ruled the state's lethal injection procedures needed overhaul.
The judge found that the state's procedures created too much risk that an inmate would suffer extreme pain while being executed. At that time, lethal injections were carried out in San Quentin's old gas chamber, which the judge found too cramped, too dark and too old for prison staff to properly administer execution drugs.
Since then, the California Department of Corrections and Rehabilitation has built a new execution chamber on the grounds of San Quentin in Northern California and made a number of changes to its procedures to address the judge's concerns.
A new federal judge has taken over the case and has not ruled on whether those changes are enough to restart executions.
Additionally, the corrections department is drafting a new set of regulations for administering lethal injections. No executions can take place until the new rules are formally adopted.