Born in eastern Montana, Jerry Kramer moved with his family from northern Utah to northern Idaho when he was in the fourth grade, settling in Sandpoint. After graduating from Sandpoint High School in 1954, he accepted a football scholarship to the University of Idaho Vandals to play for new coach Skip Stahley. In that era, Idaho was a member of the Pacific Coast Conference, the forerunner of the Pac-10.
Kramer was a standout player for the Vandals, along with teammate (and roommate) Wayne Walker of Boise, a future All-Pro linebacker with the Detroit Lions. Both played in the East-West Shrine Game and the College All-Star Game that summer, in which they defeated the defending NFL champion Lions. Kramer would have his uniform number 64 retired by the university.
Later, Jerry was enshrined in the East-West Shrine Game Hall of Fame and is also Idaho's first ever football All-American.