If you have a wife/girlfriend with you, Bon Ami is a pretty good brunch spot that my wife loves to go to. It's just off I-55. Overall favorite though is Another Broken Egg Cafe in Ridgeland. It's awesome. Several more, but those are some of my favorites?
ETA: I just noticed you said Downtown, so these may not be helpful. I don't spend any time downtown, so maybe someone else can help with that. It may be worth the drive to eat at one of the Ridgeland places though, unless you are wanting something really close.