Budget is important too, but I think budget is less important than recruiting base in terms of program potential.
Your budget and your ability to raise money drives facilities and coaching pay, both which help with recruiting, but ultimately you're still limited by your recruiting base. Of course, the programs with the better recruiting bases are the ones that have the larger populations to draw from normally, and the programs with the larger population base to draw from usually are the larger schools with the larger alumni base and walk on fanbase, which means they're also the ones with the larger budgets.
That's why LSU is such a great program. The stadium, facilities, and salary all come as part of the support they have, but the recruiting base is much more important to their success than their support.
Louisiana has almost double the NFL talent pool as Mississippi, and they don't have to split that talent with another in-state program. When you do the numbers, that makes them pretty much 4 times as likely to land an NFL draft pick with every player they sign in comparison to Ole Miss or MSU.
That's a HUGE advantage. To overcome that, you have to have a coach that can not only out-evaluate and out-recruit the schools with the better talent bases, but you have to have one that can out-scheme them too. The problem there is that if you land one of those coaches that can do all of that, he's more than likely going to have a shot to get that easier job with more pay.
You can maximize your potential by making shrewd hires and running your athletic department in the best possible fashion, but ultimately recruiting base is the biggest limiting factor in my opinion, for any program.