You're not exactly correct on that. We only played them 12 times in Vaught's 24 seasons as head coach, and I know they didn't count as a conference game every one of those 12 years. We did play Arkansas almost every year as an OOC team, and they were a very good program back then. We also played Houston, Tulane, and Memphis State fairly often in our OOC schedule.
Tulane and Memphis sucked, while Houston became a power in the mid-60's.
As for Bama, they wouldn't play us in Oxford for a while. They wanted Tuscaloosa only, which is the main reason we didn't play them for many years. We finally got them on the schedule for each of Vaught's last 6 seasons. And we played LSU in Baton Rouge 18 times in Vaught's 24 seasons. We only played them in Oxford 3 times and 3 times in Jackson.
Pretty convenient, to miss out on playing them in your "glory years" of 1954-63.
Back then the SEC didn't have any scheduling policy, and they didn't control the schedule, so the richer programs (like Bama and LSU) could treat fellow conference members like small guys and play them only at home if they wanted. Also, it didn't guarantee that every school would get the same number of conference games scheduled.
Florida wasn't good back then. Georgia Tech left the conference in 1964.
You still didn't play them in your "glory years"...Florida wasn't the power they are now, but they were still a good team.
From looking at our schedules, it looks like Vanderbilt, Kentucky, MSU, Tennessee, and LSU were the most common SEC opponents we had from those years. Auburn, Alabama, Georgia, and Florida all played us at times, but it wasn't as often, because they weren't forced to play us by the conference.