I'm from New Albany. Although I haven't lived there in 15 years, my parents still live there so I go from time to time.
George's ... It's just a fried chicken and burger place. But it's a good local place.
Taylor's ... I'm only familiar with the lunch buffet. My parents--and much of the rest of the town--go there most Sundays for lunch. If you do go on Sunday, go early. Can get pricey if you're paying for several.
West Side BBQ ... I grew up on this. But I've had much better since I've "grown up". It's still decent, but the pulled pork is nothing special. Like someone else said, their cakes are good.
Speaking of cakes, there's a good bakery downtown called Sugaree's.
Tallahatchie Gourmet opened within the last few years. I've only been a couple times but it's not bad.
If you really want a unique local stop, try Latham's. They make doughburgers. It's sorta hidden on the backside of downtown. It's been forever since I've been, and I really don't remember if I liked it. But it's unique. 98% of urbanspoon reviewers can't be wrong, right?