How to Find Competitors’ Keywords For Your SEO & PPC
Want to find out competitors’ keywords? Here’s how to find what keywords competitors use on their sites – and protect your own!
Step 1: Look at their sitemaps. This is a page, usually created for search engines, where every valuable page on a site is linked to using the keywords the page is targeting. In technical terms, the anchor text of those links is keyword rich.
Step 2: You can either:
- Copy-paste the code on that page into a file. Then use CTRL+F to find all the links starting with the first one and then copy paste those into an excel file. That excel file will then show you your competitors’ main SEO keywords.
- Or you could get a web spider, and have it crawl the page then extract the links for you. Again, this will show you competitors’ main SEO keywords.
The limits on this technique:
- It doesn’t tell you the relative value of keywords. Linda Bustos of Elastic Path Ecommerce asked me, “Should I target body wash or shower gel? Your technique doesn’t answer that.”
- It doesn’t tell you if a keyword is successful at all, or if it was just an idea that a competitor had. This really matters for keywords that are vague or ambigious. Eg “bass” – Do most people care about bass fish or bass guitars?
- For PPC advertisers, it doesn’t tell you how a competitor is pitching searchers. You’re better off knowing their most successful ‘ad copy’ (the text of their ad) rather than any copy you may have seen.
Solution (in principle): In advertising the longer an ad runs, the more likely it is to be making money for the advertiser. Similarly, the more competitors buying particular ad space, the more valuable that space probably is.
So how do you find out how valuable a keyword (the ad space) is, and how to pitch searchers? How do you know how long a competitor has been buying a keyword (the value of the space) and what their best ad copy is?
Solution (in practice): Spyfu, a competitive intelligence based keyword research tool, gives you this data. It does this with its Domain History, Ad History and Spyfu Kombat tools.
Domain History tells you what keywords a competitor bought over the past year, and for how long. By seeing the keywords they’ve bought for the longest time, you can see what keywords are most valuable.
Spyfu’s Ad History shows you all competitors who have been buying a keyword for the past year, and for how long. And it shows you their ad copy, too.
So if you see other businesses in your market buying a keyword and staying with it, it’s probably a good keyword for you. And by seeing the patterns in competitors ads, especially over time, you can see how to pitch the people searching.
I also like Spyfu for other reasons. I’ve bought several reports from Spyfu (monthly subscriptions start at $59.95, or yearly for $399) and can vouch for their value. One of the things I really love about them is not just the specific keywords they show, but finding keyword patterns or categories I hadn’t discovered during my own keyword research.
For example, I found that years were often attached as modifier keywords to car model names, when people wanted to buy, sell, or get car insurance.
Another way in which Spyfu helps you estimate keywords’ value is that you get the position their ad appeared in the search results. The higher the ad’s position, the more a competitor is bidding for that keyword and the more traffic they want on that keyword.
Spyfu value-maximization tip 1: With smaller competitors whose PPC strategy may not be very sophisticated, Spyfu doesn’t work as well. Their sweet spot is mega campaigns with hundreds or preferable thousands of keywords. So start out by finding keywords from sites like eBay or Amazon or whoever is the leader in your vertical. I.e. the people who are #1 (in either PPC or SEO) for “cars” or “shoes” or “insurance.”
Spyfu value-maximization tip 2: Look at trends in position over time. If you see a competitor gradually raising their ad’s position, chances are that the keyword is proving better than expected. And if they’re dropping, it’s probably a poor keyword!
Compete Inc offers you similar keyword data. Their collection methodology is more reliable, though because it comes directly from the search engine users! The downside is that it’s a lot more expensive than Spyfu, starting at $199/month.
Compete has toolbars on 2 million people’s computers that track their internet usage and their search engine usage in particular.
So you can see the actual keywords that are referring traffic to given websites, and which websites are getting the greatest CTR from the SERPs.
So you can “Build better search campaigns by understanding the competition,” as Compete says, which I think is a fair positioning statement since their data is looking at traffic stats as opposed to what another SEO/SEM is trying to do (my technique and Spyfu).
How do you protect your own keywords then?
While I’m no expert in SiteMaps, it’s my understanding that you can submit an XML sitemap without having a separate html sitemap on your website. If that’s the case, I would avoid making the sitemap if getting your keywords ripped off is a concern.
If you must have an html SiteMap on your site to submit one to the SEs, then this is where breaking usability convention is allowed. Hide it 5 clicks/folders deep and link to it with misleading anchor text like “ugly fat bearded ladies.” And give each folder users need to navigate to get there 5 other folders, each of them titled something really helpful, like 1,2,3,4,5…
So to get to the site map, the person would have to know the “combination” to your “folder lock” e.g. 4-2-5-4-1. You get bonus points for doing this all in Ajax – which doesn’t load a new page each time you click a link – so that they can’t use the URL to locate themselves unless they finally reach the site map.
While any decent SEO can find some of your keywords, you can at least make it a lot more difficult and laborious if they try to take shortcuts by swiping your keywords. Besides, there’s no guarantee they’ll find all the keywords you’re targeting, on their own. If you can keep them out of at least some of your markets/keywords, you’re preserving profit margins there.
Note: by observing the navigation elements of a site, competitors can find out some of your keywords anyways. Nevertheless, it’ll be very time-consuming and the competition may not be thorough enough to find all your keywords. If it’s being done by scrapers, you can try and detect them and cloak for the scrapers to not find what they needed.
Another way to protect your own keywords is to keep in mind how Spyfu and Compete work. They give information about particular domains. So if you split your PPC campaigns amongst multiple domains, your competitors would have to know all your domain names to get all your keywords. This can minimize the risk of losing all your keywords to a competitor in one swoop. Of course, that might be offset if one draws much higher CTR, but it’s a delicate balance…
Experienced direct marketers will tell you that 80% of their success lies in the ‘list’ of prospects they send their mail/email to. In search marketing your list consists of your keywords. So go find out competitors’ keywords!
I hate to bash on someone I don’t know in the SEO world, but I have to say, this is probably some of the worst advice I’ve read regarding sitemaps. Please take this as constructive criticism, as I think your readers deserve to know more about this.
Your sitemaps (plural, because you should have BOTH an XML and HTML sitemap for your site) need to be the easiest link a spider can find. Why? Because this is your one and only way to guarantee all of your pages, no matter how deep they are within your site, get indexed. By creating an intricate path of folders you are cutting yourself down significantly, especially because a sitemap isn’t really that useful for doing competitive keyword research since you only get 1 keyword per page.
An alternate suggestions for competitive keyword research is to use the search engines against your competitor. If you know they are doing a lot of link building, use link:www.DOMAIN.com to see all of the inbound links your competitor has. Then, go to those sites and see what anchor text is used to lead to the site.
This is by no ways the only or even best way to do keyword research, but it is a better alternative to scouring sitemaps in my opinion.
Reply
Comment by Kyle Wegner — May 4, 2008 @ 8:54 am
Hey Kyle,
First of all, I appreciate the criticism of the idea as opposed to criticism of my person. Constructive debate of that nature is always welcome.
It all depends on your perspective. If you build your site architecture properly to begin with, SEs shouldn’t have a problem finding all your pages and indexing them. Webmasters got large sites fully indexed before the advent of sitemaps, and therefore it must still possible afterwards.
If you’re using it to get deep pages indexed that wouldn’t get found using your site’s regular navigation, it might be worth asking whether you don’t have a usability/design issue. Alternatively, if you don’t need it to get other pages besides the deep ones indexed, perhaps a partial map might be better to avoid revealing your strategy.
Your alternative suggestion is pretty good too. I’d suggest it’s perhaps best for picking up modifiers which couldn’t all be fit into the site’s navigational anchor text, and thus complements my tactics above. I note in passing that rather than do this manually, it can be done by using SEOmoz’s toolset.
As to the intricate path of folders, that may be my own error due to my newb level knowledge of sitemaps. The point was that if you HAVE to have one on the site to correspond to your submission through Webmaster Central / MSN/ Yahoo’s equivalents, then you should bury it far and away from the prying eyes of humans. Perhaps linking to it from inauspicious/low-traffic places like the privacy policy or other TOS type statements might be better.
Reply
Comment by Gabriel Goldenberg — May 4, 2008 @ 12:57 pm
This is a neat technique Gab – I usually go look at the inbound link anchor texts, on site navigation, and also the footer navigation. Many optimized sites will have a ‘fat footer’ of sorts, leaving all the keyword anchors at the bottom of the site as a site wide internal link.
Cheers — Dev
Reply
Comment by Dev Basu — May 5, 2008 @ 3:31 pm
Glad to hear my comment was taken light heartedly Gab. I think in the end, my takeaway here is that reading posts like these makes me realize just how much I love SEO. It is an art, not a science, and every person will use their brush differently. Like art, there is good art and bad art, but most of that comes down to the viewer…in the end, if an artist has talent the value will be realized.
You have just gained a new subscriber
Reply
Comment by Kyle Wegner — May 6, 2008 @ 6:20 pm
Dev – sweet tips dude, thanks for sharing. Added some link love.
Kyle, I quite agree. There’s more than one way to skin the cat. And thanks for subscribing!
Both of you – if you care to email me your MSN messenger addresses, I’ll be happy to add you so we can chat regularly
Reply
Comment by Gabriel Goldenberg — May 7, 2008 @ 12:02 am
No offense man but this is some pretty bad advice. You want your site map to be prominent so the spiders (not to mention the visitor) can find all your pages. You never want to have it drill down into buried folders or use numbers as your anchor text. The methods you explain defeat all the benefits for having a sitemap both from a user and SEO perspective.
Not sure I agree copying others sitemaps for keywords is a great idea either because 99% of sites don’t optimize their sitemaps and anyone with a few hours could probably build a better list themselves. Maybe it would help in the research process but I doubt it because the chances of finding a well optimized sitemap in your exact industry would take more time then its worth.
Reply
Comment by Phil Laboon — May 7, 2008 @ 1:18 am
yeah its really difficult to prottect your own keywords when at the same time you have to get your competitors KW
Reply
Comment by amelia — May 7, 2008 @ 2:28 am
Sorry to come a rain on the parade… but say you had a site where you had done this, I would likely just use a “sitemap generator” to spider the site and give me a nice list of urls. In fact, xenu link sleuth would likely be the best option for that.
Additionally, you can get software such as webceo or something similar to wander around a website and make a nice list of all the keywords used etc.
Anyway, this is nice to stop the noobs from copying what you are doing, and any seo agency worth its salt will know how to get around it in moments… though I would also argue that they need not even do that as they should be fine doing their own keyword research and optimising a site on a ‘per site’ basis
Anyway, the folder lock thing is interesting as an item on its own and I enjoy your writing style. Fine tune to topic of your posts a bit more and you will go far
Thank you for the read!
Dave
Reply
Comment by David — May 7, 2008 @ 4:35 am
David, the point about using a sitemap generator is really bright – totally hadn’t thought of that. Great way to get other people’s keywords! And I didn’t know you could use Xenu that way.
As to WebCEO, I’ve tried it before and it couldn’t pick out my keywords for the life of it. Crummy piece of software imho.
Well, I suppose you’ve poked some holes in my ideas on protecting your keywords, but evidently you’ve shown that my technique for getting them from competitors works
.
Happy to hear you enjoyed the folder lock thing. I thought it was a clever trick my friend showed me.
Reply
Comment by Gabriel Goldenberg — May 7, 2008 @ 1:34 pm
Very interesting, but not too worried. If I check Google Analytics on one site it shows 17,604 key phrases in the last month. Only those at the top of the list would show up in sitemaps. Only about the top ten would show in remote anchor links. And those at the bottom of the list that don’t show are the ones that convert.
Reply
Comment by Stephan Miller — May 7, 2008 @ 2:31 pm
If you want to see a cooler pay tool which has a free trial check out Adgooroo it will tell you both organic and PPC terms. And how your market penetration is compared to your competitors. Little bit pricey but if your spending a lot on SEO, or PPC then this may save you some money (their advice is very good too).
Reply
sasdaman Reply:
July 29th, 2009 at 11:57 am
Thanks for the Adgooroo mention! Will check that out. I’m just deciding whether to go with SpyFu or KeywordSpy!? You got experience of any of the above!?
Reply
Gabriel Goldenberg Reply:
July 29th, 2009 at 12:05 pm
I’ve used and like SpyFu.
Reply
Comment by Daryl from Quenet SEO — May 10, 2008 @ 1:41 am
Stephan, you have an excellent point about this being largely limited to the shorttail. That said, the longtail tends to be composed of the shorttail plus modifiers, so this is still a valid starting point. Furthermore, I’ve done analytics on sites earning 5 figures/month from single word searches…
Daryl, thanks for sharing that. FYI, Spyfu does that and I believe Compete does too.
Reply
Comment by Gabriel Goldenberg — May 11, 2008 @ 1:13 pm
Nice work, this article made WebProNews too
Reply
Comment by Linda Bustos — May 12, 2008 @ 7:24 pm
Hey Gab! Thanks for the link love sir. Check your email, as I’m looking for your MSN/gtalk id — Dev
Reply
Comment by Dev Basu — May 15, 2008 @ 10:08 pm
You can also use Wordze’s competitor keywords tool.
http://www.wordze.com/competitorkeywords.php
And follow it up by using this tool of theirs: http://www.wordze.com/relatedkeywords.php
But there always seems to be debate over how accurate wordze is compared to other keyword research tools for some reason.
Reply
Comment by Will — May 22, 2008 @ 1:10 am
uh oh, my competition better watch out!
Reply
Comment by failureblogger — May 25, 2008 @ 5:55 pm
[...] IP lookups a la ‘my IP neighbors” won’t find all your sites in one fell swoop. Find out competitors’ keywords for [...]
Pingback by » Guest Post By SEO Superstar: Gab Goldebnberg | Utah Search Engine Optimization | Utah SEO Consultant — July 12, 2008 @ 3:58 pm
[...] http://seoroi.com/se…le-protecting-yours/ [...]
Pingback by Can You Spot 3 Little Log Details That Mean A LOT? | SEO ROI Services — January 12, 2009 @ 10:10 pm
[...] How To Find Competitors’ Keywords (For Free) [...]
Pingback by My Most Popular Posts — January 28, 2009 @ 5:52 pm
[...] through his blog SEO ROI and is particularly interested in Social Media Analytics as well as finding out competitors keywords. I had the opportunity to interview Gab about his business in SEO and he offered some great [...]
Pingback by Interview with Gab Goldenberg, 21 Year Old SEO Consultant of SEO ROI — February 16, 2009 @ 12:38 pm
There are many tools out there which profess to be the end all in keyword research. If chasing the competition was all you had in mind, this is fine. The point is to lead and formulate new thoughts and ideas while embracing the old. Why chase when you can lead.
Reply
Gabriel Goldenberg Reply:
June 23rd, 2009 at 2:10 pm
Excellent point there Dave. The reason is that Buy comes before sell, generally speaking.
Reply
Comment by Dave — June 23, 2009 @ 7:56 am
[...] check against competitors’ keywords. Are you on the right track or really, really [...]
Pingback by SEO ROI » 8 Short Steps To Forecast and Estimate SEO ROI… — August 20, 2009 @ 6:56 pm
Nice tips Gab.I’ve never tried it before and I never knew that you can protect your keywords.I thought hiding keywords is against the google guidelines.BTW,thanks I found these topic.
Reply
Gabriel Goldenberg Reply:
October 13th, 2009 at 2:19 am
It’s a question of not making the location obvious to humans … that doesn’t necessarily imply hiding it.
Reply
Comment by Andre Colt — October 12, 2009 @ 10:29 pm