Many animals, including a number of species of birds, feed their young pre-digested food. The young animals cannot break down the food's molecules enough themselves to gain access to the nutrition, so the adults do it for them. Without the pre-digested food, the young would starve.

Search engines cannot gain access to many web pages, commonly including dynamic e-commerce product level pages, leaving these pages to starve from lack of traffic. However, the practice of Search Engine Optimization can provide workarounds that help get content in front of the search engine's spiders, feeding them great amounts of data that the search engines can then effectively use in response to search queries.

It may seem like an unappetizing analogy, but it is an apt one.

Fortunately, there are a couple of ways to get inaccessible content to search engine spiders. One common method is to remove spider blocks that prevent spiders from doing their crawl of the web site, such as undertaking the project of dynamic URL re-writing. One such approach is called a Mod Rewrite, a process that removes problematic symbols in the URLs that may have prevented a spider from accessing the content for fear of a spider trap. Some webmasters undertake the Mod Rewrite themselves, essentially republishing their product or information pages in a static format that is easier for spiders to crawl effectively. Others outsource the project to a Search Engine Marketing (SEM) firm. Still others look to a specialized technology firm whose solution may include placing pages on their own server and charging you for traffic on a pay-per-click basis. While this temporarily provides spiders access to your product pages, when you stop paying, the traffic stops, too. More importantly, you haven't resolved the core issue - the inaccessibility of your web site to the spiders of the most important search engines, leaving you perennially at the starting block while your competition moves forward.

Keep in mind that removing the spider blocks will not influence how high in search rankings any particular page will appear; it will provide the spiders a pathway to access your pages so that they may now be included in the search engine database. Applying all the techniques of Search Engine Optimization (SEO) will be required for the pages to perform at their best in rankings and, thus, in traffic generation. But at least removing the spider blocks, or applying the Mod Rewrite workaround, will get you on the playing field.

Another common way to help a search engine include all the pages of your site in its database is to give a "direct feed" of your site to the search engines. Currently, Yahoo, AltaVista and HotBot, through the Yahoo Search Marketing Solutions (formerly called Overture) Site Match Xchange program, invite you or your SEM agency to submit a feed of your entire web site, including an online catalog, into its database, removing the engines' need to gain access to these pages through spidering. You are then charged on a cost-per-click basis, determined by your category of product. To start this program, webmasters or their SEMs can provide Yahoo a tab-delimited file of their products that fulfills certain criteria, including required fields, or they can build an XML file to upload to the engines. A few SEMs have developed their own spiders, spiders that can crawl even challenging client sites to relieve the client's IT team from the need to build a feed in-house. Then the SEM spider crawl, which has grabbed all the content on the client's web pages, can be packaged and optimized before being sent to the Yahoo program -- pre-spidered and highly digestible.

Note once again that just giving a feed to the Yahoo SiteMatch Xchange program does not guarantee traffic, just inclusion in the database. However, a well-constructed feed has an excellent chance at performing well and delivering an attractive and consistent R.O.I.

Unfortunately, Google, MSN and other, smaller engines do not participate in a Direct Feed program at this time, leaving the first option of removing spidering obstacles and including Search Engine Optimization techniques and the Pay-for-Placement programs as your only alternatives for keeping your site from traffic starvation from these engines at this time. But even when you have both of these elements of Search Engine Marketing working for you, you should still consider adding the SiteMatch Xchange program because it provides two significant benefits:

• You can upload a fresh feed or get re-spidered frequently, ensuring that your newest products are included in the search engines quickly and that changes in pricing or inventory are up to date at all times;
• Your guaranteed inclusion in these important search engines provide cushioning against the uncontrollable aspects of your "organic" Search Engine campaign, such as when Google changes its algorithms and from cost spikes in the Pay-for-Placement engines, such as when a competitor starts a bidding war.

Whichever techniques you choose, do all that you can to help the search engines do their job of gathering all the most important information on the Web into their gigantic databases. The results will be more than worth it, even if it means pre-spidering your website for them.

What is your point of view? Please post your thoughts on the discussion board.

Recent Posts:

Warning: main(/web/sites/www.witi.com/htdocs/users/discussion_board/conf_global.php) [function.main]: failed to open stream: No such file or directory in /system/web/lib/applications/discussion_board/htdocs/users/discussion_board/ssi.php on line 79