You make a good point, and you even see that now with the 12 team league and the two divisions.
We rotate teams in the East, outside of the one permanent for football. For basketball, you play your division twice and the others once. For baseball, the only teams that rotate off every few years is a team from the other division.
So you already play some members significantly more than others. Familiarity breeds contempt, or in this case rivalries. If you were to ask all the SEC fans to list 1-11 the teams in the league they hate the most, most of them would have the teams from their division at the top, with exceptions of traditional rivalries like UT-Bama, Georgia-Auburn that take precedent over many of the division rivals.
Personally, I hate the other 5 West schools way more than the East schools already, just because we don't cross paths with them as often. If you expand to 14, 16, or 20, you'll really water down the rivalries. If we did go to 20 teams, I guarantee it would turn into basically two 10 team leagues. We'd feel like we were a part of our 10 team league, but we would feel more like cousins with the other half of the league than actual sharing members.
Something that hasn't been discussed is the fact that a 16 team league is not unprecedented. The WAC was a 16 team league for a few years before 8 of the schools chose to split off and create the MWC. I think it would be wise for Slive and his cronies to people that were involved in that league to get an idea what the issues were. I felt like it was way too big at the time. Granted, it wasn't a power conference, which made it easier to collapse, but I think the 16 team model is risky. That said, I also think it's risky to not make a move and get left behind in the movement.
I'd like to see the NCAA change the rules so that 16 team leagues could have a 2 round championship playoff for the conference title. If they'd do that, you could split the league into 4 divisions, essentially like the NFC and AFC and have each division winner fight it out. That would help with the issue of schedule rotation, because you'd only have 3 permanent opponents and could rotate the other 5 or 6 slots on the conference slate.