Google Helps Web Sites Get Crawled -- for FreeKnown for its regular introduction of new products as much as for its regularly increasing stock price, Google is continually evolving. Recently the Internet giant unveiled another worthwhile product, Google Sitemaps, aimed at what it calls an attempt to expand its coverage of the World Wide Web and increase the freshness of its indexed Web pages.
The free service lets Webmasters place an XML-formatted file on their servers so Googlebot can better crawl all Web pages on a site. The new offering also lets Webmasters make updates to the Sitemap when changes to the Web site have been made so that the Google spiders will know to crawl the new content as well. At its best, this would mean that no Web page is left behind and possibly lead to increased search relevancy -- a benefit for searchers and the search industry, in general -- and better search placement for participating Web sites.
Google could be using this feature to play catch-up after a recent "study" of 10,000 rated searches, conducted by RustyBrick to test which engine (Google, Yahoo Search, MSN Search or Ask Jeeves) returned the most relevant results. So far, Yahoo is beating Google by a small margin. But when it comes to the biggest search engine in the world losing out to its main competitor, even a small margin can become a big problem.
Google Sitemaps became part of the discussion at last week's WebmasterWorld Search and Marketing Conference in New Orleans. One thousand attendees -- including myself -- were present to witness a not-so-friendly exchange of words between Google (Matt Cutts) and Yahoo (Tim Mayer) representatives over -- what else? -- search engine rankings. One good point made by Google engineer Cutts was that "Google Sitemaps is like paid inclusion for free." Yahoo, the only search engine to offer paid inclusion, allows a Web site to pay for spiders to conduct regular visits to the site, better ensuring that all, or at least the most important, pages on a site will be scanned and have a better chance to appear higher in the search engine results.
Yahoo has received some negative feedback for offering this service, with many claiming that paid inclusion basically lets Web sites pay for placement in regular search results, therefore skewing the whole purpose of a search engine -- to provide the most relevant, unbiased search results.
And though some could argue that the new Google Sitemaps potentially may skew search results because it is being offered free of charge to anyone, someone else could argue equally well that the new service will actually help ensure better search relevancy because it lets any Web site have its pages regularly updated, even if just for the purpose of "keeping up with the Joneses."