Pawel Szulencki Search Engine Optimization/Marketing blog.
If you're new here, you may want to subscribe to my RSS feed or follow me on Twitter. Thanks for visiting!
I wanted to check how fast are search engines in crawling the web. After publishing blog post “How to move WordPress blog from old domain to new domain with 301 permanent redirection” i went to Google, Yahoo! and MSN to check how fast can they find that new post.
It was kind of search engines speed test, like the one you do to test your bandwidth speed but instead of your bandwidth i tested how fast search engines find information on Internet.
Google was really fast in this matter. When i checked it after only 3 hours, the post was already there! I have published the post on August 6, 2008 at 1:00 pm and when i checked at 4:30 pm on Google the post was there already for 30 minutes!
Yahoo!
After the same period of time Yahoo! showed no results.
The Yahoo! search results message says:
We did not find results for: “How to move WordPress blog from old domain to new domain with 301 permanent redirection”. Try the suggestions below or type a new query above.
MSN
MSN was no better. It also returned similiar not found message:
We did not find any results for “How to move WordPress blog from old domain to new domain with 301 permanent redirection”.
Conclusion
Google is much faster in picking changes on blogs than other search engines. Maybe it’s because they have separate Google blog search engine which is specially designed to keep an eye on blog changes. Other search engines don’t have special blog search engines specialised to find new posts/pages or changes within them.
Anyway Google is really fast nowadays in searching for new web pages on the Internet.
After 7 hours of waiting for Yahoo! or MSN to pick up the new post, still nothing and i give up. Maybe my website doesn’t have enough reputation for those search engines to index it more frequently. So how good news from a website like mine is supposed to reach users as fast as possible? Only through Google.
[Update August 11, 2008] After 4 days since publishing my post and checking the search engines for the first time i have checked again Yahoo! and MSN. To my surprise Yahoo! showed just 2 results (both from my website) and MSN showed 30 results but NONE of them was from my blog!
Do you have any experience in search engines crawling speed? Share your discoveries.
Sphere: Related ContentPawel Szulencki is a SEO (Search Engine Optimization) and Marketing certified specialist who is interested in organic SEO, paid campaigns (PPC) and Social Media Marketing channels. (Read more)
limeshot (2 comments.)
August 13th, 2008 at 9:59 am
Hi Pawel,
I found that I can get a blog entry indexed in 15 minutes flat. Google is getting really good at indexing blog entries since they introduced the Ping service and I think most blogging platforms will ping the Google bot automatically, as soon as a new item is published.
Pawel Szulencki (18 comments.)
August 13th, 2008 at 10:25 am
@limeshot,
15 minutes is really nice time. When you have sth interesting and valuable to say the time is precious.
Yes, pinging makes it easier, but why, ohh, why doesn’t Yahoo! and MSN do the same thing? Don’t they care about bloggers?
limeshot (2 comments.)
August 15th, 2008 at 3:19 pm
I actually heard of a chap who could get his posts indexed in 11 minutes. I think it varies, but Wordpress does a good job of pinging various indexes - amongst which yahoo, google blogsearch, technorati, feedburner, etc. Why yahoo and msn don’t come to visit even after pinging is unclear to me apart from the fact that Google made a mission out of promoting fresh content; which the other two engines have not.
Pawel Szulencki (18 comments.)
August 15th, 2008 at 8:25 pm
Good point limeshot. Google is trying to promote fresh content and to achieve that they have to be fast in picking that new content on the web.
And 11 minutes as you said, wow, thats fast. Maybe in the nearby feature search engines (not only Google) will be able to index websites live, as they change and reflect it in their indexes. Crazy idea?