Because you are getting the wrong visitors. You have a bucket and spade shop but you are nowhere near the beach! Your bounce rate (how fast people leave your site) is probably very high.
You should act immediately because there may be a huge opportunity waiting for you. If you do not the search engines will soon pick up the way users are backing out and they will drop you from their results
What to do:
Or in other words, stop selling buckets and spades and start selling sandwiches!
I'm Tim Hill, a Search Engine Optimisation and Online Marketing specialist. I created this site to help others understand that SEO is not a mysterious black art!.
If your a newbie try the Getting Started in SEO page, otherwise feel free to dig around and learn more.