Many animals, including some species of birds, feed their young pre-digested food. The young animals cannot break down the food’s molecules enough by themselves to gain access to the nutrition, so the adults do it for them. Without the pre-digested food, the young would starve.
Search engines cannot gain access to many Web pages, commonly including dynamic e-commerce product-level pages, leaving these pages to starve from lack of traffic. However, the practice of search engine optimization can provide workarounds that help get content in front of the spiders, feeding them great amounts of data the engines can use in response to search queries. It may seem like an unappetizing analogy, but it is an apt one.
There are a couple ways to get inaccessible content to search engine spiders. One common method is to remove spider blocks that prevent spiders from doing their crawl of the Web site, such as undertaking the project of dynamic URL rewriting. One such approach is called a Mod Rewrite, a process that removes problematic symbols in the URLs that may have prevented a spider from accessing the content for fear of a spider trap.
Some Webmasters undertake the Mod Rewrite themselves, essentially republishing their product or information pages in a static format that is easier for spiders to crawl effectively. Others outsource the project to a search engine marketing firm while some look to a specialized technology firm whose solution may include placing pages on its own server and charging you for traffic on a pay-per-click basis. Though that temporarily gives spiders access to your product pages, when you stop paying, the traffic stops, too. More importantly, you haven’t resolved the core issue: the inaccessibility of your site to the spiders of the most important search engines, leaving you perennially at the starting block while your competition moves forward.
Remember that removing spider blocks will not influence how high in search rankings any particular page will appear. It will provide spiders a pathway to access your pages so that they now may be included in the search engine database. Search engine optimization techniques are required for the pages to perform their best in rankings and, thus, in traffic generation. But at least removing the spider blocks, or applying the Mod Rewrite workaround, gets you on the playing field.
Another common way to help a search engine include all the pages of your site in its database is to give a “direct feed” of your site to the engine. Yahoo, AltaVista and HotBot – through the Yahoo Search Marketing Solutions (formerly called Overture) SiteMatch Xchange program – invite you or your SEM agency to submit a feed of your entire Web site, including an online catalog, into its database, removing the engines’ need to gain access to these pages through spidering. You then are charged on a cost-per-click basis determined by your category of product.
To start this program, Webmasters or their SEMs can provide Yahoo a tab-delimited file of their products that fulfills certain criteria, including required fields, or they can build an XML file to upload to the engines. A few SEMs have developed their own spiders, spiders that can crawl even challenging client sites to relieve the client’s IT team from the need to build a feed in-house. Then the SEM spider crawl can be packaged and optimized before being sent to the Yahoo program – pre-spidered and highly digestible.
Note once again that just giving a feed to the Yahoo SiteMatch Xchange program does not guarantee traffic, just inclusion in the database. However, a well-constructed feed has an excellent chance to perform well and deliver an attractive, consistent ROI.
Google, MSN and other, smaller engines do not participate in a direct feed program. This leaves the first option of removing spidering obstacles and including SEO techniques and the pay-for-placement programs as your only alternatives for keeping your site from traffic starvation from these engines at this time. But even when you have both of these elements of search engine marketing working for you, still consider adding the SiteMatch Xchange program because it provides two benefits:
• You can upload a fresh feed or get re-spidered frequently, ensuring that your newest products are included in the search engines quickly and that changes in pricing or inventory are up to date at all times.
• Your guaranteed inclusion in these important engines provides cushioning against the uncontrollable aspects of your “organic” search engine campaign, such as when Google changes its algorithms, and from cost spikes in the pay-for-placement engines, such as when a competitor starts a bidding war.
