How to Find Competitors’ Keywords And Protect Your Own
Want to find out competitors’ keywords? Want to avoid getting your keyword research ripped off? Here’s how to find competitive keyword intelligence for free!
Look at their site maps. Every valuable page on their site will be linked to from there, with desirable anchor text. You can either:
- Copy-paste the code on that page into a file, then use CTRL+F to find all the links starting with the first one and then copy paste those into an excel file.
- Or you could get a free web spider, and have it crawl the page then extract the links for you.
Linda Bustos of Elastic Path Ecommerce pointed out to me that by using my Sitemap scraping technique, you don’t know which keywords are most valuable. Linda gave this example question to illustrate: “Should we target shower gels or body wash?” That’s an excellent observation. The answer is to use PPC to find out what keywords convert before optimizing for organic rankings.
Spyfu is awesome and very affordable for swiping PPC keywords. Not only do you get your competitors’ list, but you also get the associated rank that Spyfu had them at when they scraped the SERP.
I’ve bought several reports from Spyfu and can vouch for their value. One of the things I really love about them is not just the specific keywords they show, but finding keyword patterns or categories I hadn’t discovered during my own keyword research.
For example, I found that years were often attached as modifier keywords to car model names, when people wanted to buy, sell, or get car insurance.
So it’s possible to make an educated guess about the most valuable keywords by looking at the ad rank of your competitors; the higher up they are, the greater value they’re probably placing on that keyword. The reason is that if you’re higher, you tend to get more clicks; ergo their ad rank tells you how much they value clicks on that keyword.
Caveat: With smaller competitors whose PPC strategy may not be very sophisticated, Spyfu doesn’t work as well. Their sweet spot is mega campaigns with hundreds or preferable thousands of keywords. So start out by finding keywords from sites like eBay or Amazon or whoever is the leader in your vertical. I.e. the people who are #1 (in either PPC or SEO) for “cars” or “shoes” or “insurance.”
Compete Inc offers you similar keyword data. Their collection methodology is more reliable, though because it comes directly from the search engine users!
Compete has toolbars on 2 million people’s computers that track their internet usage and their search engine usage in particular.
So you can see the actual keywords that are referring traffic to given websites, and which websites are getting the greatest CTR from the SERPs.
So you can “Build better search campaigns by understanding the competition,” as Compete says, which I think is a fair positioning statement since their data is looking at traffic stats as opposed to what another SEO/SEM is trying to do (my technique and Spyfu).
How do you protect your own keywords then?
While I’m no expert in SiteMaps, it’s my understanding that you can submit an XML sitemap without having a separate html sitemap on your website. If that’s the case, I would avoid making the sitemap if getting your keywords ripped off is a concern.
If you must have an html SiteMap on your site to submit one to the SEs, then this is where breaking usability convention is allowed. Hide it 5 clicks/folders deep and link to it with misleading anchor text like “ugly fat bearded ladies.” And give each folder users need to navigate to get there 5 other folders, each of them titled something really helpful, like 1,2,3,4,5…
So to get to the site map, the person would have to know the “combination” to your “folder lock” e.g. 4-2-5-4-1. You get bonus points for doing this all in Ajax - which doesn’t load a new page each time you click a link - so that they can’t use the URL to locate themselves unless they finally reach the site map.
While any decent SEO can find some of your keywords, you can at least make it a lot more difficult and laborious if they try to take shortcuts by swiping your keywords. Besides, there’s no guarantee they’ll find all the keywords you’re targeting, on their own. If you can keep them out of at least some of your markets/keywords, you’re preserving profit margins there.
Note: by observing the navigation elements of a site, competitors can find out some of your keywords anyways. Nevertheless, it’ll be very time-consuming and the competition may not be thorough enough to find all your keywords. If it’s being done by scrapers, you can try and detect them and cloak for the scrapers to not find what they needed.
Another way to protect your own keywords is to keep in mind how Spyfu and Compete work. They give information about particular domains. So if you split your PPC campaigns amongst multiple domains, your competitors would have to know all your domain names to get all your keywords. This can minimize the risk of losing all your keywords to a competitor in one swoop. Of course, that might be offset if one draws much higher CTR, but it’s a delicate balance…
Experienced direct marketers will tell you that 80% of their success lies in the ‘list’ of prospects they send their mail/email to. In search marketing your list consists of your keywords. So go find out competitors’ keywords!
I hate to bash on someone I don’t know in the SEO world, but I have to say, this is probably some of the worst advice I’ve read regarding sitemaps. Please take this as constructive criticism, as I think your readers deserve to know more about this.
Your sitemaps (plural, because you should have BOTH an XML and HTML sitemap for your site) need to be the easiest link a spider can find. Why? Because this is your one and only way to guarantee all of your pages, no matter how deep they are within your site, get indexed. By creating an intricate path of folders you are cutting yourself down significantly, especially because a sitemap isn’t really that useful for doing competitive keyword research since you only get 1 keyword per page.
An alternate suggestions for competitive keyword research is to use the search engines against your competitor. If you know they are doing a lot of link building, use link:www.DOMAIN.com to see all of the inbound links your competitor has. Then, go to those sites and see what anchor text is used to lead to the site.
This is by no ways the only or even best way to do keyword research, but it is a better alternative to scouring sitemaps in my opinion.
[Reply]
Comment by Kyle Wegner — May 4, 2008 @ 8:54 am
Hey Kyle,
First of all, I appreciate the criticism of the idea as opposed to criticism of my person. Constructive debate of that nature is always welcome.
It all depends on your perspective. If you build your site architecture properly to begin with, SEs shouldn’t have a problem finding all your pages and indexing them. Webmasters got large sites fully indexed before the advent of sitemaps, and therefore it must still possible afterwards.
If you’re using it to get deep pages indexed that wouldn’t get found using your site’s regular navigation, it might be worth asking whether you don’t have a usability/design issue. Alternatively, if you don’t need it to get other pages besides the deep ones indexed, perhaps a partial map might be better to avoid revealing your strategy.
Your alternative suggestion is pretty good too. I’d suggest it’s perhaps best for picking up modifiers which couldn’t all be fit into the site’s navigational anchor text, and thus complements my tactics above. I note in passing that rather than do this manually, it can be done by using SEOmoz’s toolset.
As to the intricate path of folders, that may be my own error due to my newb level knowledge of sitemaps. The point was that if you HAVE to have one on the site to correspond to your submission through Webmaster Central / MSN/ Yahoo’s equivalents, then you should bury it far and away from the prying eyes of humans. Perhaps linking to it from inauspicious/low-traffic places like the privacy policy or other TOS type statements might be better.
[Reply]
Comment by Gabriel Goldenberg — May 4, 2008 @ 12:57 pm
This is a neat technique Gab - I usually go look at the inbound link anchor texts, on site navigation, and also the footer navigation. Many optimized sites will have a ‘fat footer’ of sorts, leaving all the keyword anchors at the bottom of the site as a site wide internal link.
Cheers — Dev
[Reply]
Comment by Dev Basu — May 5, 2008 @ 3:31 pm
Glad to hear my comment was taken light heartedly Gab. I think in the end, my takeaway here is that reading posts like these makes me realize just how much I love SEO. It is an art, not a science, and every person will use their brush differently. Like art, there is good art and bad art, but most of that comes down to the viewer…in the end, if an artist has talent the value will be realized.
You have just gained a new subscriber
[Reply]
Comment by Kyle Wegner — May 6, 2008 @ 6:20 pm
Dev - sweet tips dude, thanks for sharing. Added some link love.
Kyle, I quite agree. There’s more than one way to skin the cat. And thanks for subscribing!
Both of you - if you care to email me your MSN messenger addresses, I’ll be happy to add you so we can chat regularly
[Reply]
Comment by Gabriel Goldenberg — May 7, 2008 @ 12:02 am
No offense man but this is some pretty bad advice. You want your site map to be prominent so the spiders (not to mention the visitor) can find all your pages. You never want to have it drill down into buried folders or use numbers as your anchor text. The methods you explain defeat all the benefits for having a sitemap both from a user and SEO perspective.
Not sure I agree copying others sitemaps for keywords is a great idea either because 99% of sites don’t optimize their sitemaps and anyone with a few hours could probably build a better list themselves. Maybe it would help in the research process but I doubt it because the chances of finding a well optimized sitemap in your exact industry would take more time then its worth.
[Reply]
Comment by Phil Laboon — May 7, 2008 @ 1:18 am
yeah its really difficult to prottect your own keywords when at the same time you have to get your competitors KW
[Reply]
Comment by amelia — May 7, 2008 @ 2:28 am
Sorry to come a rain on the parade… but say you had a site where you had done this, I would likely just use a “sitemap generator” to spider the site and give me a nice list of urls. In fact, xenu link sleuth would likely be the best option for that.
Additionally, you can get software such as webceo or something similar to wander around a website and make a nice list of all the keywords used etc.
Anyway, this is nice to stop the noobs from copying what you are doing, and any seo agency worth its salt will know how to get around it in moments… though I would also argue that they need not even do that as they should be fine doing their own keyword research and optimising a site on a ‘per site’ basis
Anyway, the folder lock thing is interesting as an item on its own and I enjoy your writing style. Fine tune to topic of your posts a bit more and you will go far
Thank you for the read!
Dave
[Reply]
Comment by David — May 7, 2008 @ 4:35 am
David, the point about using a sitemap generator is really bright - totally hadn’t thought of that. Great way to get other people’s keywords! And I didn’t know you could use Xenu that way.
As to WebCEO, I’ve tried it before and it couldn’t pick out my keywords for the life of it. Crummy piece of software imho.
Well, I suppose you’ve poked some holes in my ideas on protecting your keywords, but evidently you’ve shown that my technique for getting them from competitors works ;).
Happy to hear you enjoyed the folder lock thing. I thought it was a clever trick my friend showed me.
[Reply]
Comment by Gabriel Goldenberg — May 7, 2008 @ 1:34 pm
Very interesting, but not too worried. If I check Google Analytics on one site it shows 17,604 key phrases in the last month. Only those at the top of the list would show up in sitemaps. Only about the top ten would show in remote anchor links. And those at the bottom of the list that don’t show are the ones that convert.
[Reply]
Comment by Stephan Miller — May 7, 2008 @ 2:31 pm
If you want to see a cooler pay tool which has a free trial check out Adgooroo it will tell you both organic and PPC terms. And how your market penetration is compared to your competitors. Little bit pricey but if your spending a lot on SEO, or PPC then this may save you some money (their advice is very good too).
[Reply]
Comment by Daryl from Quenet SEO — May 10, 2008 @ 1:41 am
Stephan, you have an excellent point about this being largely limited to the shorttail. That said, the longtail tends to be composed of the shorttail plus modifiers, so this is still a valid starting point. Furthermore, I’ve done analytics on sites earning 5 figures/month from single word searches…
Daryl, thanks for sharing that. FYI, Spyfu does that and I believe Compete does too.
[Reply]
Comment by Gabriel Goldenberg — May 11, 2008 @ 1:13 pm
Nice work, this article made WebProNews too
[Reply]
Comment by Linda Bustos — May 12, 2008 @ 7:24 pm
Hey Gab! Thanks for the link love sir. Check your email, as I’m looking for your MSN/gtalk id — Dev
[Reply]
Comment by Dev Basu — May 15, 2008 @ 10:08 pm
You can also use Wordze’s competitor keywords tool.
http://www.wordze.com/competitorkeywords.php
And follow it up by using this tool of theirs: http://www.wordze.com/relatedkeywords.php
But there always seems to be debate over how accurate wordze is compared to other keyword research tools for some reason.
[Reply]
Comment by Will — May 22, 2008 @ 1:10 am
uh oh, my competition better watch out!
[Reply]
Comment by failureblogger — May 25, 2008 @ 5:55 pm
[...] IP lookups a la ‘my IP neighbors” won’t find all your sites in one fell swoop. Find out competitors’ keywords for [...]
Pingback by » Guest Post By SEO Superstar: Gab Goldebnberg | Utah Search Engine Optimization | Utah SEO Consultant — July 12, 2008 @ 3:58 pm
[...] http://seoroi.com/se…le-protecting-yours/ [...]
Pingback by Can You Spot 3 Little Log Details That Mean A LOT? | SEO ROI Services — January 12, 2009 @ 10:10 pm
[...] How To Find Competitors’ Keywords (For Free) [...]
Pingback by My Most Popular Posts — January 28, 2009 @ 5:52 pm
[...] through his blog SEO ROI and is particularly interested in Social Media Analytics as well as finding out competitors keywords. I had the opportunity to interview Gab about his business in SEO and he offered some great [...]
Pingback by Interview with Gab Goldenberg, 21 Year Old SEO Consultant of SEO ROI — February 16, 2009 @ 12:38 pm
There are many tools out there which profess to be the end all in keyword research. If chasing the competition was all you had in mind, this is fine. The point is to lead and formulate new thoughts and ideas while embracing the old. Why chase when you can lead.
[Reply]
Gabriel Goldenberg Reply:
June 23rd, 2009 at 2:10 pm
Excellent point there Dave. The reason is that Buy comes before sell, generally speaking.
[Reply]
Comment by Dave — June 23, 2009 @ 7:56 am