The biggest thing for me is the picture quality of the TV that I want to watch. The TV on my HDTV TiVo is just breathtaking so I'm kind of spoiled.
This morning I headed up to the Apple store to check out the picture quality for myself and was very disappointed. On the demo model it had there (and I'm assuming as a demo at its store it's using the best possible TV and video content it can get) the TV picture quality was just not good enough for me. It's not HDTV quality.
It was playing this "terra" channel thing as part of the demo and although it was "DiscoveryHD like" the picture quality is much worse. Maybe this picture quality is "good enough" for most people, but it's not "good enough" for me when compared to the picture quality I get on my HDTV TiVo.
TV picture quality is a subjective thing of course, but I'd encourage anyone who is considering buying an Apple TV to first head down to an Apple store and see the picture quality in a best case scenario yourself and see if you think it's good enough for you before plunking down $300 on this thing.