Too many webmasters end up on a wild goose chase because they think they have been penalized and attempt to repair the wrong thing.
Some even get themselves in hot water by submitting a reconsideration request when Google was not aware of any issues or using the disavow tool which flags up to Google that you have spammy links when it was not aware!
In essence they self report themselves, instigating a manual review which can throw up far more issues and even lead to bans.
You have only been penalized by Google if you have received a message in Google Webmaster Tools to that effect (see below).
If you have received no such message your drop in rankings may be down to:
In almost all manual penalties Google will tell you if the penalty is 'site wide' or 'partial'.
Site wide means the whole of your website has been hit by the penalty and the whole (or nearly all) of your website is guilty of the specified action.
Partial means that only part of the website has been hit by the penalty. You may, for example, have a blog and a forum. The blog is untouched by the penalty but the forum is hit hard. This is Google's way of saying "We can see there is more than one aspect to this website and recognise only one particular part/aspect is guilty".
Before trying to recover from a Google manual penalty first check that you actually have one. In Google Webmaster Tools go to Search Traffics -> Manual Actions and look for a message relating to one of the following: