Depends on what part of town you're in. Downtown and points to the east and north, no problem. To the south and west, maybe you'll catch some guff.
I'm a UK fan and alumnus living in Louisville and there are just as many UK fans as UofL fans. We're just not as obnoxious.