Understanding the competition should be a key component of planning
your SEO. The first step is to understand who your competitors in the
search results really are. It can often be small players who give you a
run for your money. For example, consider the previously mentioned credit
card search in Google; Visa, Master Card, American Express, and Discover
Card all fail to reach the #1 position in the Google results.
Instead, you will find these results dominated by affiliate players.
Affiliates tend to be the most adept at search engine optimization and can
be the most lax in abiding by the search engines’ terms and
1. Two Spam Examples
Affiliates that cheat tend to come and go out of the top search
results, as only sites that implement ethical tactics are likely to
maintain their positions over time. You can help expedite the cheaters’
fall from grace by reporting them to Google at http://www.google.com/contact/spamreport.html, or better
yet, via the dashboard in your Google Webmaster Tools account (where it
will carry more weight).
How do you know whether a top-ranking site is playing by the
rules? Look for dubious links to the site using a backlink analysis tool
such as Linkscape.
Since the number of links is one factor search engines use to determine
search position, less ethical websites will obtain links from a
multitude of irrelevant and low-quality sites.
This sort of sleuthing can reveal some surprises. For instance,
here are examples of two devious link schemes:
GiftCertificates.com’s short-lived nemesis was FindGiftCards.com, which came
out of nowhere to command the top two spots in Google for the
all-important search term gift certificates,
thus relegating GiftCertificates.com to
the third position. How did FindGiftCards.com do it? It
operated a sister site, 123counters.com, with a free
hit counter that propagated “link spam” across thousands of sites,
all linking back to FindGiftCards.com and other
sites in its network.
Sadly for FindGiftCards.com, Stephan
Spencer, founder and president of the e-marketing agency
Netconcepts, outed the company in an article he wrote for
Multichannel Merchant back in 2004 (http://multichannelmerchant.com/catalogage/ar/marketing_casing_competition/),
and Google became aware of the scam. The end result? Down to two
pages in the Google index, as shown in Figure 1.
CraigPadoa.com was a thorn in
the side of SharperImage.com, outranking
the latter for its most popular product, the Ionic Breeze, by
frameset trickery and guestbook spamming (in other words, defacing
vulnerable websites with fake guestbook entries that contained
spammy links back to its own site). As soon as The Sharper Image
realized what was happening, it jumped on the wayward affiliate. It
also restricted such practices in its affiliate agreement and
stepped up its monitoring for these spam practices.
Figure 1. Site with only two pages in the index
2. Seeking the Best
Look for competitors whose efforts you would like to emulate (or
“embrace and extend,” as Bill Gates would put it)—usually a website that
consistently dominates the upper half of the first page of search
results in the search engines for a range of important keywords that are
popular and relevant to your target audience. Note that your “mentor”
competitors shouldn’t just be good performers; they should also
demonstrate that they know what they’re doing when it comes to SEO. To
assess competitors’ competence at SEO, you need to answer the following
Are their websites fully indexed by Google and Yahoo!? In
other words, are all their web pages, including product pages,
making it into the search engines’ databases? You can use the free
tool at http://www.netconcepts.com/urlcheck to
find out, or you can go to each search engine and type in
site:theirdomain.com. A competitor with only a
small percentage of its site in Google probably has a site that is
unfriendly to search spiders.
Do their product and category pages have keyword-rich page
titles (title tags) unique to each page? You can easily review an
entire site’s page titles within Google or Yahoo! by searching for
Incidentally, this type of search can sometimes yield
confidential information. A lot of webmasters do not realize that
Google has discovered and indexed commercially sensitive content
buried deep in its site. For example, a Google search for
confidential business plan filetype:doc will
yield a lot of real business plans among the sample
Do their product and category pages have reasonably high
Is anchor text across the site, particularly in the
Are the websites getting penalized? You can overdo SEO. Too
much keyword repetition or too many suspiciously well-optimized text
links can yield a penalty for over-optimization. Sites can also be
penalized for extensive amounts of duplicate content.
Are they spamming the search engines with “doorway
3. Uncovering Their Secrets
Let’s assume your investigation has left you with several
competitors who are gaining excellent search placement using legitimate,
intelligent tactics. Now it is time to uncover their secrets:
What keywords are they targeting? You can determine this by
looking at the page titles (up in the blue bar at the top of your
web browser, which also appears in the search results listings)
across their home page and product category pages, then by looking
at their meta keywords tag (right-click, select View Source, and
then scour the HTML source for the list of keywords that follow the
bit of HTML that looks something like the following:
<meta name="keywords" content="keyword1, keyword2, …">
Who’s linking to their home page, or to their top-selling
product pages and category pages? A link popularity checker can be
quite helpful in analyzing this. Amazon.com has been especially
successful in achieving deep links into its millions of
If it is a database-driven site, what technology tricks are
they using to get search engine spiders such as Googlebot to cope
with the site being dynamic? Nearly all the technology tricks are
tied to the e-commerce platforms the competitors are running. You
can check to see whether they are on the same server software as you
by using the “What’s that site running?” tool on the top-left corner
of http://www.netcraft.com. Figure 2 shows a screen shot of the
results for HSN.com.
While you are at it, look at “cached” (archived) versions of
your competitors’ pages by clicking on the Cached link next to their
search results in Google to see whether they’re doing anything too
aggressive, such as cloaking, where they serve
up a different version of the page to search engine spiders than to
human visitors. The cached page will show you what the search engine
actually saw, and you can see how it differs from the page you see
when you go to the web page yourself.
What effect will their future SEO initiatives have on their
site traffic? Assess the success of their SEO not just by the lift
in rankings. Periodically record key SEO metrics over time—the number of pages indexed, the
PageRank score, the number of links—and watch the resulting effect
on their site traffic.
You do not need access to competitors’ analytics data or
server logs to get an idea of how much traffic they are getting.
Simply go to Compete,
Quantcast, or Alexa, and search on the
competitor’s domain. If you have the budget for higher-end
competitive intelligence tools, you can use Compete or Hitwise.
The data these tools can provide is limited in its accuracy,
but still very useful in giving you a general assessment of where
they are. The tools are most useful when making relative comparisons
between sites in the same market space. To get an even better idea,
use their capabilities to compare the traffic of multiple sites. In
this mode, you can get a pretty accurate idea as to how your traffic
compares to theirs.
You can now get this type of data directly from Google as
well, using Google
Trends for Websites. The output of this tool is just a
summary of Google traffic, but it is a much larger data set than is
available from the other products. Figure 3 shows an example of the
output from Google Trends for Websites.
Note that tools such as Alexa, Compete, and Quantcast do have
other unique features and functionality not available in Google
Trends for Websites.
How does the current state of their sites’ SEO compare with
those of years past? You can reach back into history and access
previous versions of your competitors’ home pages and view the HTML
source to see which optimization tactics they were employing back
then. The Wayback
Machine provides an amazingly extensive archive of web
Figure 2. Sample Netcraft output
Figure 3. Google Trends for Websites