Getting back on G track that simple...
 

So, is it as simple as removing all those shitty directory links on blogs and resubmitting to google? Or is it much more than that?

That's one way, and it does help. Especially if the directories have rolled off your links depending on how old they are.

Looking at prior traffic trends, sadly most have only delivered 50-100 hits in total. Doing so for the link back was worth it but once it falls off page one its just junk imo now.