Just as you can be penalized for having unnatural incoming links to your website so to can you be penalized for hosting unnatural links to other websites.
This is often the hit which takes down directories but will affect your pages if you seem to link to completely unrelated topics. If I were to, for example, suddenly include a link in the middle of this text to "Holidays in Florida" that would indeed be unnatural.
In fact if I were to include links like "Holidays in Florida" anywhere on this page it would be out of context with the page's content and so look suspicous. Surely the only reason I would put such links in is if I was being paid to do it and if so I should have marked the links with the rel="NO FOLLOW" tag.
A one off is not enough to receive a penalty. I might have the link "Holidays in Florida" in order to link to a certain point. I would need to be carrying out this activity on a large scale as members of link wheels often do.
Examples of where webmasters often get hit include:
In most cases, however, this penalty is aimed at link farms and content farms who will dump the domain once they've been hit, having made what money they could from hopeful webmasters.
The best move is to remove the links completely. If you placed them in there to "help someone out" by passing some Page Rank through to them then adding the 'NO FOLLOW' tag to those links will stop the links helping so there is not much point keeping them.
If they are paid links make sure whoever is paying knows you will be changing them to 'NO FOLLOW' links and then they can decide whether or not to continue with them.
Document everything, page by page and with dates so you can submit a convincing reconsideration report to Google.