Born in Port Arthur, Texas, Johnson graduated from Thomas Jefferson High School (now Memorial High School) in Port Arthur, where one of his classmates was future rock superstar Janis Joplin, whom Johnson nicknamed "beat weeds."
He went to college at the University of Arkansas and was a member of the 1964 National Championship football team, where he was an all-SWC defensive lineman for Hall of Fame coach Frank Broyles, and a teammate of future Dallas Cowboys owner Jerry Jones. Other notable teammates were Ken Hatfield, Jim Lindsey, All-American and Ernie Davis Award Winner, Ronnie Caveness, and future Outland Trophy winner Loyd Phillips. Several future great head coaches were assistant coaches for Frank Broyles and the Razorbacks during Johnson's career in Fayetteville: Hayden Fry, future legendary Head Coach at the University of Iowa, Johnny Majors, future legendary Head Coach at the University of Tennessee, and most notably Barry Switzer, Hall of Fame coach of the University of Oklahoma and the man who replaced Jimmy Johnson as the head coach of the Dallas Cowboys. Johnson was nick-named "Jimmy Jumpup" because he never stayed down on the ground for long during football practices or games as it was said his determination was boundless.