Unfortunately the size and display of something that looks good on a computer monitor looks very different on a 320 pixel wide display. This is why the best approach is to maintain a standard and a mobile specific version. There are enough people surfing on mobile phones now to justify this. While the OP doesn't mind it, a lot of people really do hate scrolling around and zooming in on a site meant for a computer screen. It's a much better experience for an end user of your site to be able to choose which version of the site they prefer to view.
2 years ago we first started offering mobile pay site versions as an option for customers using the Elevated X CMS. Since then I've never had a single user ask me about a universal version instead. I've also heard nothing but positive feedback from customers who have mobile versions of their sites.
A separate, mobile optimized versions is the way to go.
AJ