Home

Although this site has been produced for specific courses and groups of students it is designed as a public resource. If you find it useful then please let me know.

If you want to comment feel free to do so and if you find something wrong get in touch.

hide alert

Why won't Googlebot come and look at Food Parcels?

Written by: Jonathan Briggs

September 13, 2005 [3191 views]

I am busy trying to get Food Parcels into the search listings and its proving harder than normal. Working with search engines is a deeply black art and sometimes the things that normally work don't.

Here are the normal rules:

  1. Write good relevant content
  2. Make sure the design is good (and simple)
  3. Use good headings, titles, link names and keywords
  4. Update regularly
  5. Encourage the spiders to find you by linking from high quality sites

It is this last piece that has not worked. For some reason, despite 20-30 carefully chosen links, Googlebot and MSNbot have stayed away.

Each morning I check the logs and I can see bots from yahoo!, askjeeves and lots of smaller search sites but no Google or MSN.

Is this important?

Of course it is. On the current web, traffic is all important and it will become more important for ecommerce sites such as Food Parcels as we move towards Christmas.

Recent comments:

On October 5, 2005 at 4:39 PM, joe wrote:

tekirdag

http://www.kanal59.com

On November 12, 2005 at 12:49 PM, Jonathan wrote:

Well, we found the answer! And it was a mixture of changes in the way we serve and the way Google and MSNbot index our sites.

It used not to matter whether you included a "robots.txt" file for your site or not. Robots.txt is normally used to tell a search engine spider that you do not want parts of your site crawled. The non-existence of this file implied that you did not mind the crawler looking everywhere. So far so good.

Well behaved spiders always check to see whether the robots.txt exists. In food parcels case it did not but this should not have mattered.

But, the way we serve our sites has changed and instead of simply reporting that the file did not exist our server reported an error. Google noticed the error and decided not to look any further. Solution - add a robots.txt file!

It has taken us weeks to find this error and it was so simple and staring us in the face. The lesson is simple; don't assume anything about the way your technology works particularly when it comes in contact with someone else's technology.

What do you think?






Add your comments

Recent postings in this section

Site map | Copyright | Credits | Contact
© copyright 2004-2012 Jonathan Briggs. Content licensed under Creative Commons

 

Sound installation at Vauxhall Ragged School as part of London Open House

Sound installation at Vauxhall Ragged School as part of London Open House [Sep 20, 2009]

Photo blog

Search

About this site

This site is an educational resource, a research project and a publication channel written by Jonathan Briggs, Professor of eCommerce at Kingston University.
As well as being Professor at Kingston, Jonathan...

Read more

Creative Commons License
This work is licenced under a Creative Commons Licence

Interesting stats

Locations of visitors to this page