Search Engine Indexing with Sitemaps

One of the most common problems ecommerce websites have with rankings is that portions of their websites are not included in the search engines. The cause can be any of a variety of reasons that range from complicated URL strings to slow server response or lack of a proper internal linking structure. If Google can’t find your pages, they won’t get into the search results.

Search engines benefit from having as many good quality pages in their databases as possible because it improves user experience. Better user experience equals more traffic and that means more advertising revenue. As a result, several efforts have been made to make it easier for webmasters to have their websites indexed more effectively. This article discusses efforts Google and Yahoo! have made to augment current methods of discovering as many URLs of a web site as possible.

First there was Google Sitemaps

Earlier this summer Google announced their Google Sitemaps tool that allows website owners to create an XML or plain text file listing all of the URLs of a web site for inclusion in the Google index. At least up to 50,000 are allowed per document and if you’re over, you can just create another site map.

Google offers this tool as a compliment to what Googlebot, Google’s search engine spider, can find on its own by crawling the web. In cases where a site might have some difficulty getting indexed, providing a sitemap to Google may assist in getting missing pages into Google’s search results.

Google does not charge for this beta service and it’s pretty easy to do. However, as a beta, there are no guarantees. More information is available on Google Sitemaps.

What about Yahoo!?

Yahoo has offered a free web site submission form for quite a while where, as a registered user, you can submit one URL at a time. There is no guarantee of inclusion and no way to know if your site is accepted except to check the Yahoo! search results for your web site.

For companies that want to provide a large number of URLs to Yahoo!, the Yahoo! Search Submit paid inclusion program offers the opportunity to have guaranteed inclusion in Yahoo!’s organic index. Search results display intermingled with URLs that are discovered naturally by Slurp, Yahoo!’s web crawler. There is a cost per URL for inclusion and also a cost per click based on the category of your business. Unlike sponsored listings, there is no effect on ranking to use the Search Submit paid inclusion program, only the guarantee that your web site will be included in the Yahoo! search engine index and that all links will be revisited every 48 hours.

You can also pay Yahoo! $299 per year to be included in the Yahoo! Directory. There is no guarantee; even if you pay, that you will be accepted or that your suggested title, description and category will be used.

And then there was Yahoo! Sitemaps

A free, bulk URL submission option for Yahoo! was first presented at the recent Search Engine Strategies conference in San Jose from Yahoo!’s Tim Mayer during a session on indexing. Previously, websites were discovered for the organic search results by the SLURP web crawler or by submitting one URL at a time. Now, Yahoo! offers the option of submitting many URLs at once using Yahoo!’s free submission service. Registration for a Yahoo! account is required, but that is free.

Here’s how it works: Create a list of all the URLs you want indexed in a text file and name it something like, urllist.txt (Yahoo!’s suggestion). You can also provide a compressed file and call it urllist.gz. Example: domainname.com/urllist.txt.

If you are providing a plain text list of URLs to Google Sitemaps, then you can use that same file for Yahoo!. Place the file in the main directory of your web site and submit the URL to the Yahoo! free submission form.

Rather than paying for inclusion into the Yahoo! index, many websites with crawlable URLs may benefit from using the Yahoo! Sitemap submission option rather than the Search Submit paid inclusion. The trade-off is, there is no guarantee of inclusion and if your site changes often, those changes will not be reflected in the search results until the site is update by the Yahoo! web crawler. With Search Submit paid inclusion, your URLs will be updated every 48 hours.

It’s important to note that inclusion is not search engine optimization. But it is important, because if none or only part of your site is included in a search engine’s index, then you could be missing out on significant amounts of natural search traffic. Submitting a sitemap to Google and Yahoo! has no known side effects and may give your site a boost in overall rankings since you’ll have more pages available to search results.