SEO: Try Surfing Like a Search Engine Spider

I’m a visual learner. I need to see and interact with things to understand them. It therefore helps me to mimic the behavior of search engine spiders when I’m digging into a site for the first time to review its search engine optimization issues. I like to think of it as “being the bot.”

The key thing to remember is that search engines crawlers don’t have JavaScript, CSS or cookies enabled. They can’t “see” content embedded in media files such as images, Flash, video or audio. You take all this away, and what are you left with? Plain HTML, text and links. In many cases, the entire focus of the page is rendered indecipherable, and in some cases the site isn’t even navigable. It’s an incredible illustration of the structural challenges bots (i.e. search engine robots) face when they’re crawling a site.

Emulate a Search Engine Spider

Firefox’s Web Developer Toolbar by Chris Pederick will make surfing like a search spider much easier. For this exercise, focus on the yellow highlighted sections of the toolbar and select the following features:

1. Disable > Disable JavaScript > All JavaScript
2. Cookies > Disable Cookies > All Cookies
3. CSS > Disable Styles > All Styles
4. Images > Disable Images > All Images
5. Visit any page and you’re surfing approximately like a bot. If you’re already at the page you want to test, reload the page to get the full experience.

Another way to confirm what Googlebot is seeing is to visit Google’s text-only cache with JavaScript, cookies and CSS disabled. It should look the same as the live page with all those things disabled. If it doesn’t, it’s possible there is some form of user agent or bot detection in play on the site to deliver a different experience to search engine crawlers.

Let’s take a test drive. It’s very easy to see the difference on many major consumer brands, because their sites tend to be dependent on Flash and JavaScript to deliver a glossy, engaging, interactive brand experience. But what does this cost organic search traffic and sales from this traffic?

Peek at Victoria’s Secret

We’ll start with Victoriassecret.com, clearly a very brand-conscious site.

For starters, the header and footer navigation require JavaScript to load, effectively removing the navigation the site for search engine spider and for visitors without JavaScript enabled.

The primary content is image-based. The only text on the page for search engine spiders to crawl is the alternative attributes in the seven feature images on the page. This handful of alt attribute text, like the images, is entirely focused on enticing humans to click. No argument, that’s an important function of a home page. However, Victoria’s Secret is keeping its keyword theme of its home page a secret from the search engines. The title tag sends the sole focused keyword signal on the page.

If we were to peek at Victoria Secrets’ web analytics, I’ll bet dollars to doughnuts that they receive the vast majority of their natural search traffic and sales for branded search phrases (such as “victorias secret”), and a very small percentage for non-branded terms (such as “lingerie”). Despite Victoria Secret’s efforts to feature dresses, swimwear and sandals on the home page, it’s highly unlikely that it’s winning any non-branded searches for those very popular summer fashion search phrases. In short, creating strong textual signals for www.victoriassecret.com and ensuring that the navigation is crawlable would be the first step to winning non-branded traffic and sales.

What About Adidas?

Now let’s review another example, Adidas.com.

Adidas is using a tactic known as graceful degradation to provide a crawlable text-based experience when the user/agent visiting doesn’t support JavaScript or CSS. It’s certainly not pretty to human eyes, but it’s not supposed to be – that’s what the whiz-bang Flash version is for. To a bot or a human on a text reader it’s perfectly navigable.

That said, Adidas should pay more attention to the kinds of text it’s providing. While the amount of text and links is strong, the template and the words used don’t send a coherent keyword theme. The home page lacks a strong title tag and H1 heading, though there are several H3 headings. The text in the title tag and the H3 headings aren’t sending a signal that Adidas is a power in shoes, athletics, sports or any other relevant keyword market that Adidas should legitimately be able to capture.

Summary

Removing the signals that we humans are trained to pay attention to – the shiny, glossy images and animations – forces us to consider the textual signals that a site is really sending on each page. If those textual signals are weak, unfocused or nonexistent, search engines will have a harder time matching your site with popular search phrases when real customers are searching for your products.