There is an obvious flaw in the OP's line of thinking. As an "intelligent civilization" we have only been able to send signals that would travel through space for about a century - which is nothing in terms of light years. And we have not even had a manned space mission to another planet in our own solar system. There could literally be billions of planets out there with comparable civilizations, but our technology is far too limited at this time to even detect them - and I won't even start about how small the budget is for S.E.T.I. and some other factors. In light of our limited abilities, how can we say the fact that we have not been visited is conclusive proof of anything?
Sometimes it helps to visualize a situation, so here is my favorite YouTube video, the Hubble Deep Field video. It helps put some perspective on the amount of other solar systems we are talking about in the entire Universe.
Hubble Deep Field: The Most Imp. Image Ever Taken (Redux) - YouTube