Another concern, the report says, is that 44 percent of school administrators report slashing big chunks of time from physical education, arts and recess since the passage of the No Child Left Behind law in 2001 in order to boost classroom time for reading and math.
With childhood obesity on the rise about 17 percent of children ages 2 through 19 are obese and kids spending much of the day in the classroom, the chairman of the committee that wrote the report said schools are the best place to help shape up the nation's children.
"Schools for years have been responsible for various health programs such as nutrition, breakfast and lunch, immunizations, screenings," Harold W. Kohl III, a professor of epidemiology at the University of Texas School of Public Health, said in an interview with The Associated Press.
"Physical activity should be placed alongside those programs to make it a priority for us as a society," he said.
The report calls on the Education Department to recommend that PE be adopted as a core subject.
It says physical education in school is the "only sure opportunity" for youngsters to have access to activity that will help keep them healthy.
The majority of states, about 75 percent, mandate PE, according to the American Alliance for Health, Physical Education, Recreation, and Dance. But most do not require a specific amount of time for PE in school, and more than half allow exemptions or substitutions, such as marching band, cheerleading and community sports.
Many kids also aren't going to gym class at school every single day. According to the CDC, only about 30 percent of students nationwide attend PE classes five days a week.
Specifically, the report recommends:
All elementary school students should spend an average of 30 minutes each day in PE class.
Middle and high school students should spend an average of 45 minutes each day in PE class.
State and local officials should find ways get children more physical activity in the school environment.
PE isn't the sole solution, though.
The report advocates a "whole-of-school" approach where recess and before-and-after-school activities including sports are made accessible to all students to help achieve the 60-minutes-a-day recommendation for physical activity. It could be as simple as having kids walk or bike to school, or finding ways to add a physical component to math and science class lessons.
The report also cautions against taking away recess as a form of punishment, and it urges schools to give students frequent classroom breaks.
Schools can do this if they make it a priority, said Paul Roetert, CEO of the American Alliance for Health, Physical Education, Recreation, and Dance.
"We have an obligation to keep kids active," Roetert said in an interview. "We have research to show that physical activity helps kids perform better in school. It helps them focus better in the classroom ... and they behave better in school. So there are all kinds of side benefits."
Kitty Porterfield, spokeswoman for The School Superintendents Association, said nobody is opposed to physical education.
"Everybody would love to see more of it in schools," said Porterfield. "Given the testing and academic pressures for excellence on schools, often physical education slides to the bottom of the barrel."
The idea of putting more of an emphasis on physical education in schools has support in Congress.
Rep. Marcia Fudge, D-Ohio, plans to introduce the PHYSICAL Act on Thursday. It would recognize health education and physical education as core subjects within elementary and secondary schools. Reps. John Lewis, D-Ga., and Jared Polis, D-Colo., will join Fudge as co-sponsors.