First of all, if you guys are treated only half as well as we were at Clemson, you will have great trip.
I agree with those suggesting staying downtown; there are a number of lodging options. Most of the "should see" stuff is downtown (H&B Museum, YUM! Center, Ali Museum, Frazier Arms Museum, etc.). Those are all "walk to" attractions if you are staying downtown. Lots of restaurants downtown ranging from pricey steakhouses and $$$$ dining spots to "high end bar food" spots, many around the YUM! Center. FWIW, the 3 classic special occasion type dining spots are The English Grill, The Oak Room (located in historic hotels) and Vincenzo's. If you are in a car, there are many good restaurants along Bardstown Road and in some of the suburban centers.
There is a casino located across the river in southern IN (Caesar's Horseshoe) but that would be car ride from downtown (I have no idea about shuttle service between downtown hotels and Horseshoe). Kentucky is famous for bourbon and if you have a car you could make a half day trip to the Jim Beam Distillery if that sort of thing appeals to you.
Since it is an easy walk from Churchill Downs (i.e., Museum, track will be dark) to the stadium I would visit there on game day. The tailgate scene is not anywhere near as attractive as Clemson but you will likely find many tailgaters who are more than willing to share a beer and a burger. You can easily cab or bus to Churchill Down/Stadium area from downtown but I honestly don't know about cabbing back in town after the game. My first thought is to walk over to the Cardinal Cafe and have a drink while the crowds clear out then call a cab from there. (At Clemson, I was parked close to the Esso Bar and spent an hour or so in there waiting for traffic to clear...and it never did!)
The Cardinal Cafe is a decent little restaurant located on the grounds of the Hilton Garden Inn. The Hilton Garden Inn is easily the best accommodation in the immediate stadium area but other than the Cardinal Cafe there is little in the way of restaurants in that entire area. If you stay at the Garden Inn the walking distance dining choices are the Cardinal Cafe, Cracker Barrel and a few fast food places.
Hope you have a great trip (except, of course, for the ball game).
Peace