OP, if you take your code and make a WP theme out of it, so that WP outputs exactly the same html code as your non-WP version, do you think it will hurt your rankings? i think not as engines will see exactly the same markup
with this we can conclude that it's not massively WP's fault, but a theme's markup and optimization. or you don't agree?