How to Steal Competitors’ Keywords While Protecting Your Own
Want to find out competitors’ keywords? Want to avoid getting your keyword research ripped off? Here’s how to do competitive keyword intelligence for free.
Look at their site maps. Every valuable page on their site will be linked to from there, with desirable anchor text. You can basically copy-paste the code on that page into a file, then use CTRL+F to find all the links starting with the first one and then copy paste those into an excel file. If you can code, there’s probably a much easier way to do this, which my friends Shady, Gabriel Malca or the bright folks at RKG might be able to tell you.
How do you protect your own then? While I’m no expert in Site Maps, it’s my understanding that you can submit one without having a separate one on your website. If that’s the case, I would avoid making the site map if getting your keywords ripped off is a concern.
If you must have a Site Map on your site to submit one to the SEs, then this is where breaking usability convention is allowed. By all means hide it 5 clicks/folders deep and with misleading anchor text like “ugly fat bearded ladies.” Oh, and each folder should have 5 folders, each of them titled something really helpful, like 1,2,3,4,5. So to get to the site map, the person would have to know the “combination” to your “folder lock” e.g. 4-2-5-4-1. Bonus points for doing this all in Ajax so that they can’t use the URL to locate themselves unless they finally reach the site map.
(The folder tip comes from a high school friend who taught me how he hid porn on his computer. I never used it, of course.)
That said, any quality ecommerce and/or ecommerce SEO operation should be able to tell you what the valuable keywords are to begin with, beginning with product names and brands. So depending on who you’re up against, keep in mind that an experienced team is going to nail the right keywords regardless. But at least you can make it a lot more difficult and laborious if they try and take shortcuts to swipe yours. Besides, there’s no guarantee they’ll figure out all the keywords you’re targeting on their own.
I note that by carefully observing the standard navigation elements of a site, that you can figure out their keywords anyways. Nevertheless, it’ll be a more time-consuming process and they may not be thorough enough to get all your keywords. If it’s being done by scrapers, you can try and detect them and cloak for the scrapers to not find what they needed.
Linda Bustos pointed out to me that by using my Sitemap scraping technique, you don’t know which keywords are most valuable. Linda gave this example question to illustrate: “Should we target shower gels or body wash?” That’s an excellent observation, which my friend Dave Davis has an answer for: use PPC to determine what converts before optimizing (he says it in the context of typos, but it works anywhere) .
Spyfu (non-aff) is awesome and very affordable for swiping PPC keywords. Not only do you get your competitors’ list, but you also get the associated rank that Spyfu had them at when they scraped the SERP.
So it’s possible to make an educated guess about the most valuable keywords by looking at the ad rank of your competitors; the higher up they are, the greater value they’re probably placing on that keyword. The reason is that if you’re higher, you tend to get more clicks; ergo their ad rank tells you how much they value clicks on that keyword.
Caveat: With smaller competitors whose PPC strategy may not be very sophisticated, Spyfu doesn’t work as well. Their sweet spot is mega campaigns with hundreds or preferable thousands of keywords.
This also goes back to the tip I’ve given before: split your PPC campaigns amongst multiple domains to avoid the risk of losing all your keywords to a competitor in one swoop. Of course, that might be offset if one draws much higher CTR, but it’s a delicate balance…
On a related note, Compete offers you similar data, with the difference that its collection is more reliable than any of those laid out above. Their data is pulled from the search engine user’s side.
Compete has toolbars on 2 million people’s computers that track their internet usage and their search engine usage in particular. So you can see the queries that are referring traffic to given websites, and which websites are getting the greatest CTR from the SERPs.
So you can “Build better search campaigns by understanding the competition,” as Compete says, which I think is a fair positioning statement since their data is looking at traffic stats as opposed to what another SEO/SEM is trying to do (my technique and Spyfu). Here’s the link to get started (for as little as $20). (Affiliate)
Experienced direct marketers will tell you that 80% of their success lies in the ‘list’ of prospects they send their mail/email to; in search marketing your list consists of your keywords. (Enjoyed this? Subscribe to my RSS feed.)
Comment Love Update: Kyle Wegner of Automaticable had some intelligent criticism of this post’s idea. I responded with a rebuttal here.
Sharing a variation on the technique is Dev Basu, a friend and Toronto SEO Consultant.
Subscribe to SEO ROI's RSS Feed
Curious why you should subscribe? Here's 20 reasons why. On a related note, if you need SEO services, here's where to get more information.
I hate to bash on someone I don’t know in the SEO world, but I have to say, this is probably some of the worst advice I’ve read regarding sitemaps. Please take this as constructive criticism, as I think your readers deserve to know more about this.
Your sitemaps (plural, because you should have BOTH an XML and HTML sitemap for your site) need to be the easiest link a spider can find. Why? Because this is your one and only way to guarantee all of your pages, no matter how deep they are within your site, get indexed. By creating an intricate path of folders you are cutting yourself down significantly, especially because a sitemap isn’t really that useful for doing competitive keyword research since you only get 1 keyword per page.
An alternate suggestions for competitive keyword research is to use the search engines against your competitor. If you know they are doing a lot of link building, use link:www.DOMAIN.com to see all of the inbound links your competitor has. Then, go to those sites and see what anchor text is used to lead to the site.
This is by no ways the only or even best way to do keyword research, but it is a better alternative to scouring sitemaps in my opinion.
Comment by Kyle Wegner — May 4, 2008 @ 8:54 am
Hey Kyle,
First of all, I appreciate the criticism of the idea as opposed to criticism of my person. Constructive debate of that nature is always welcome.
It all depends on your perspective. If you build your site architecture properly to begin with, SEs shouldn’t have a problem finding all your pages and indexing them. Webmasters got large sites fully indexed before the advent of sitemaps, and therefore it must still possible afterwards.
If you’re using it to get deep pages indexed that wouldn’t get found using your site’s regular navigation, it might be worth asking whether you don’t have a usability/design issue. Alternatively, if you don’t need it to get other pages besides the deep ones indexed, perhaps a partial map might be better to avoid revealing your strategy.
Your alternative suggestion is pretty good too. I’d suggest it’s perhaps best for picking up modifiers which couldn’t all be fit into the site’s navigational anchor text, and thus complements my tactics above. I note in passing that rather than do this manually, it can be done by using SEOmoz’s toolset.
As to the intricate path of folders, that may be my own error due to my newb level knowledge of sitemaps. The point was that if you HAVE to have one on the site to correspond to your submission through Webmaster Central / MSN/ Yahoo’s equivalents, then you should bury it far and away from the prying eyes of humans. Perhaps linking to it from inauspicious/low-traffic places like the privacy policy or other TOS type statements might be better.
Comment by Gabriel Goldenberg — May 4, 2008 @ 12:57 pm
This is a neat technique Gab - I usually go look at the inbound link anchor texts, on site navigation, and also the footer navigation. Many optimized sites will have a ‘fat footer’ of sorts, leaving all the keyword anchors at the bottom of the site as a site wide internal link.
Cheers — Dev
Comment by Dev Basu — May 5, 2008 @ 3:31 pm
Glad to hear my comment was taken light heartedly Gab. I think in the end, my takeaway here is that reading posts like these makes me realize just how much I love SEO. It is an art, not a science, and every person will use their brush differently. Like art, there is good art and bad art, but most of that comes down to the viewer…in the end, if an artist has talent the value will be realized.
You have just gained a new subscriber
Comment by Kyle Wegner — May 6, 2008 @ 6:20 pm
Dev - sweet tips dude, thanks for sharing. Added some link love.
Kyle, I quite agree. There’s more than one way to skin the cat. And thanks for subscribing!
Both of you - if you care to email me your MSN messenger addresses, I’ll be happy to add you so we can chat regularly
Comment by Gabriel Goldenberg — May 7, 2008 @ 12:02 am
No offense man but this is some pretty bad advice. You want your site map to be prominent so the spiders (not to mention the visitor) can find all your pages. You never want to have it drill down into buried folders or use numbers as your anchor text. The methods you explain defeat all the benefits for having a sitemap both from a user and SEO perspective.
Not sure I agree copying others sitemaps for keywords is a great idea either because 99% of sites don’t optimize their sitemaps and anyone with a few hours could probably build a better list themselves. Maybe it would help in the research process but I doubt it because the chances of finding a well optimized sitemap in your exact industry would take more time then its worth.
Comment by Phil Laboon — May 7, 2008 @ 1:18 am
yeah its really difficult to prottect your own keywords when at the same time you have to get your competitors KW
Comment by amelia — May 7, 2008 @ 2:28 am
Sorry to come a rain on the parade… but say you had a site where you had done this, I would likely just use a “sitemap generator” to spider the site and give me a nice list of urls. In fact, xenu link sleuth would likely be the best option for that.
Additionally, you can get software such as webceo or something similar to wander around a website and make a nice list of all the keywords used etc.
Anyway, this is nice to stop the noobs from copying what you are doing, and any seo agency worth its salt will know how to get around it in moments… though I would also argue that they need not even do that as they should be fine doing their own keyword research and optimising a site on a ‘per site’ basis
Anyway, the folder lock thing is interesting as an item on its own and I enjoy your writing style. Fine tune to topic of your posts a bit more and you will go far
Thank you for the read!
Dave
Comment by David — May 7, 2008 @ 4:35 am
David, the point about using a sitemap generator is really bright - totally hadn’t thought of that. Great way to get other people’s keywords! And I didn’t know you could use Xenu that way.
As to WebCEO, I’ve tried it before and it couldn’t pick out my keywords for the life of it. Crummy piece of software imho.
Well, I suppose you’ve poked some holes in my ideas on protecting your keywords, but evidently you’ve shown that my technique for getting them from competitors works ;).
Happy to hear you enjoyed the folder lock thing. I thought it was a clever trick my friend showed me.
Comment by Gabriel Goldenberg — May 7, 2008 @ 1:34 pm
Very interesting, but not too worried. If I check Google Analytics on one site it shows 17,604 key phrases in the last month. Only those at the top of the list would show up in sitemaps. Only about the top ten would show in remote anchor links. And those at the bottom of the list that don’t show are the ones that convert.
Comment by Stephan Miller — May 7, 2008 @ 2:31 pm
If you want to see a cooler pay tool which has a free trial check out Adgooroo it will tell you both organic and PPC terms. And how your market penetration is compared to your competitors. Little bit pricey but if your spending a lot on SEO, or PPC then this may save you some money (their advice is very good too).
Comment by Daryl from Quenet SEO — May 10, 2008 @ 1:41 am
Stephan, you have an excellent point about this being largely limited to the shorttail. That said, the longtail tends to be composed of the shorttail plus modifiers, so this is still a valid starting point. Furthermore, I’ve done analytics on sites earning 5 figures/month from single word searches…
Daryl, thanks for sharing that. FYI, Spyfu does that and I believe Compete does too.
Comment by Gabriel Goldenberg — May 11, 2008 @ 1:13 pm
Nice work, this article made WebProNews too
Comment by Linda Bustos — May 12, 2008 @ 7:24 pm
Hey Gab! Thanks for the link love sir. Check your email, as I’m looking for your MSN/gtalk id — Dev
Comment by Dev Basu — May 15, 2008 @ 10:08 pm
You can also use Wordze’s competitor keywords tool.
http://www.wordze.com/competitorkeywords.php
And follow it up by using this tool of theirs: http://www.wordze.com/relatedkeywords.php
But there always seems to be debate over how accurate wordze is compared to other keyword research tools for some reason.
Comment by Will — May 22, 2008 @ 1:10 am
uh oh, my competition better watch out!
Comment by failureblogger — May 25, 2008 @ 5:55 pm