How to Find Competitors’ Keywords For Your SEO & PPC

Want to find out competitors’ keywords? Here’s how to find what keywords competitors use on their sites – and protect your own!

There are three ways to find out what keywords competitors your are trying to rank for or buy with PPC ads. Click any link to skip to that technique.

Technique 1: The Competitor Site Analysis

Technique 2: Buy That Data and More From A Keyword Monitoring Service

Technique 3: Buy The Data From A Panel Measurement Service

Competitor Site Analysis

Step 1: Look at their sitemaps. This is a page, usually created for search engines, where every valuable page on a site is linked to using the keywords the page is targeting. In technical terms, the anchor text of those links is keyword rich.

Step 2: You can either:

  • Copy-paste the code on that page into a  file. Then use CTRL+F to find all the links starting with the first one and then copy paste those into an excel file. That excel file will then show you your competitors’ main SEO keywords.
  • Or you could get a web spider, and have it crawl the page then extract the links for you. Again, this will show you competitors’ main SEO keywords.

This technique has limits, though. It doesn’t:

  • Show the relative value of keywords. Linda Bustos of Elastic Path Ecommerce asked me, “Should I target body wash or shower gel? Your technique doesn’t answer that.”
  • Show the absolute value of a keyword. Does it convert, at all? Imagine an ambiguous keyword. Eg “bass” – Is that bass fish or bass guitars?
  • Tell you what competitors’ message is, in their PPC or SEO listing. Are they promoting a deal or upmarket? Special feature or ease of use?

Solution (in principle): Find out how many competitors and how long competitors have been buying a keyword. If many of them have been advertising on it for a while, it’s effective.

Spyfu can do tell you that…

Technique 2: Buy The Data From A Keyword Monitoring Service, such as Spyfu

Spyfu Competitive Intelligence Spy on Competitors AdWords and Keywords

Spyfu has 2 tools that answer the limits of the analysis we ran of our competitors’ website.

1 – Spyfu Domain History tells you what keywords a competitor bought over the past year, and for how long. By seeing the keywords they’ve bought for the longest time, you can see what keywords are most valuable.

2 -Ad History shows you all the competitors who have been bidding on a keyword in PPC auctions for the past year, and for how long they’ve been bidding.

Ad History shows you their ad text, too. By seeing the patterns in competitors ads, especially over time, you can see how to pitch the people searching.

I also like Spyfu for other reasons. Spyfu helps find keyword patterns or categories I hadn’t discovered during my own keyword research. It seems years are often used with car model names, when people want to get car insurance.

Spyfu also tells you the competitors’ ad ranking in the search results. The higher the ad’s position, the more that traffic is important to them. A good indicator of value…

Spyfu value-maximization tip Look at trends in position over time. If you see a competitor gradually raising their ad’s position, chances are that the keyword is proving better than expected. And if they’re dropping, it’s probably a poor keyword!

Compete Inc offers you similar keyword data. Their collection methodology is more reliable, though because it includes data from a “panel” – a group of 2 million people. Their actual use of the search engines is tracked! (The downside is that it’s a lot more expensive than Spyfu, starting at $199/month.)

With Compete you can see the actual keywords that are referring traffic to given websites, and which websites are getting the greatest CTR from the SERPs.
So you can “Build better search campaigns by understanding the competition,” as Compete says, which I think is a fair positioning statement since their data is looking at traffic stats as opposed to what another SEO/SEM is trying to do (my technique and Spyfu).

How do you protect your own keywords then?

While I’m no expert in SiteMaps, it’s my understanding that you can submit an XML sitemap without having a separate html sitemap on your website. If that’s the case, I would avoid making the sitemap if getting your keywords ripped off is a concern.

If you must have an html SiteMap on your site to submit one to the SEs, then this is where breaking usability convention is allowed. Hide it 5 clicks/folders deep and link to it with misleading anchor text like “ugly fat bearded ladies.” And give each folder users need to navigate to get there 5 other folders, each of them titled something really helpful, like 1,2,3,4,5…

So to get to the site map, the person would have to know the “combination” to your “folder lock” e.g. 4-2-5-4-1. You get bonus points for doing this all in Ajax – which doesn’t load a new page each time you click a link – so that they can’t use the URL to locate themselves unless they finally reach the site map.

While any decent SEO can find some of your keywords, you can at least make it a lot more difficult and laborious if they try to take shortcuts by swiping your keywords. Besides, there’s no guarantee they’ll find all the keywords you’re targeting, on their own. If you can keep them out of at least some of your markets/keywords, you’re preserving profit margins there.

Note: by observing the navigation elements of a site, competitors can find out some of your keywords anyways. Nevertheless, it’ll be very time-consuming and the competition may not be thorough enough to find all your keywords. If it’s being done by scrapers, you can try and detect them and cloak for the scrapers to not find what they needed.

Another way to protect your own keywords is to keep in mind how Spyfu and Compete work. They give information about particular domains. So if you split your PPC campaigns amongst multiple domains, your competitors would have to know all your domain names to get all your keywords. This can minimize the risk of losing all your keywords to a competitor in one swoop. Of course, that might be offset if one draws much higher CTR, but it’s a delicate balance…

Experienced direct marketers will tell you that 80% of their success lies in the ‘list’ of prospects they send their mail/email to. In search marketing your list consists of your keywords. So go find out competitors’ keywords!

Tags: , ,

Comments

  1. I hate to bash on someone I don't know in the SEO world, but I have to say, this is probably some of the worst advice I've read regarding sitemaps. Please take this as constructive criticism, as I think your readers deserve to know more about this. Your sitemaps (plural, because you should have BOTH an XML and HTML sitemap for your site) need to be the easiest link a spider can find. Why? Because this is your one and only way to guarantee all of your pages, no matter how deep they are within your site, get indexed. By creating an intricate path of folders you are cutting yourself down significantly, especially because a sitemap isn't really that useful for doing competitive keyword research since you only get 1 keyword per page. An alternate suggestions for competitive keyword research is to use the search engines against your competitor. If you know they are doing a lot of link building, use link:www.DOMAIN.com to see all of the inbound links your competitor has. Then, go to those sites and see what anchor text is used to lead to the site. This is by no ways the only or even best way to do keyword research, but it is a better alternative to scouring sitemaps in my opinion.

    Comment by Kyle Wegner - May 4, 2008 @ 8:54am
  2. Hey Kyle, First of all, I appreciate the criticism of the idea as opposed to criticism of my person. Constructive debate of that nature is always welcome. It all depends on your perspective. If you build your site architecture properly to begin with, SEs shouldn't have a problem finding all your pages and indexing them. Webmasters got large sites fully indexed before the advent of sitemaps, and therefore it must still possible afterwards. If you're using it to get deep pages indexed that wouldn't get found using your site's regular navigation, it might be worth asking whether you don't have a usability/design issue. Alternatively, if you don't need it to get other pages besides the deep ones indexed, perhaps a partial map might be better to avoid revealing your strategy. Your alternative suggestion is pretty good too. I'd suggest it's perhaps best for picking up modifiers which couldn't all be fit into the site's navigational anchor text, and thus complements my tactics above. I note in passing that rather than do this manually, it can be done by using SEOmoz's toolset. As to the intricate path of folders, that may be my own error due to my newb level knowledge of sitemaps. The point was that if you HAVE to have one on the site to correspond to your submission through Webmaster Central / MSN/ Yahoo's equivalents, then you should bury it far and away from the prying eyes of humans. Perhaps linking to it from inauspicious/low-traffic places like the privacy policy or other TOS type statements might be better.

    Comment by Gabriel Goldenberg - May 4, 2008 @ 12:57pm
  3. This is a neat technique Gab - I usually go look at the inbound link anchor texts, on site navigation, and also the footer navigation. Many optimized sites will have a 'fat footer' of sorts, leaving all the keyword anchors at the bottom of the site as a site wide internal link. Cheers -- Dev

    Comment by Dev Basu - May 5, 2008 @ 3:31pm
  4. Glad to hear my comment was taken light heartedly Gab. I think in the end, my takeaway here is that reading posts like these makes me realize just how much I love SEO. It is an art, not a science, and every person will use their brush differently. Like art, there is good art and bad art, but most of that comes down to the viewer...in the end, if an artist has talent the value will be realized. You have just gained a new subscriber :)

    Comment by Kyle Wegner - May 6, 2008 @ 6:20pm
  5. Dev - sweet tips dude, thanks for sharing. Added some link love. Kyle, I quite agree. There's more than one way to skin the cat. And thanks for subscribing! Both of you - if you care to email me your MSN messenger addresses, I'll be happy to add you so we can chat regularly

    Comment by Gabriel Goldenberg - May 7, 2008 @ 12:02am
  6. No offense man but this is some pretty bad advice. You want your site map to be prominent so the spiders (not to mention the visitor) can find all your pages. You never want to have it drill down into buried folders or use numbers as your anchor text. The methods you explain defeat all the benefits for having a sitemap both from a user and SEO perspective. Not sure I agree copying others sitemaps for keywords is a great idea either because 99% of sites don't optimize their sitemaps and anyone with a few hours could probably build a better list themselves. Maybe it would help in the research process but I doubt it because the chances of finding a well optimized sitemap in your exact industry would take more time then its worth.

    Comment by Phil Laboon - May 7, 2008 @ 1:18am
  7. yeah its really difficult to prottect your own keywords when at the same time you have to get your competitors KW :)

    Comment by amelia - May 7, 2008 @ 2:28am
  8. Sorry to come a rain on the parade... but say you had a site where you had done this, I would likely just use a "sitemap generator" to spider the site and give me a nice list of urls. In fact, xenu link sleuth would likely be the best option for that. Additionally, you can get software such as webceo or something similar to wander around a website and make a nice list of all the keywords used etc. Anyway, this is nice to stop the noobs from copying what you are doing, and any seo agency worth its salt will know how to get around it in moments... though I would also argue that they need not even do that as they should be fine doing their own keyword research and optimising a site on a 'per site' basis :) Anyway, the folder lock thing is interesting as an item on its own and I enjoy your writing style. Fine tune to topic of your posts a bit more and you will go far :) Thank you for the read! Dave

    Comment by David - May 7, 2008 @ 4:35am
  9. David, the point about using a sitemap generator is really bright - totally hadn't thought of that. Great way to get other people's keywords! And I didn't know you could use Xenu that way. As to WebCEO, I've tried it before and it couldn't pick out my keywords for the life of it. Crummy piece of software imho. Well, I suppose you've poked some holes in my ideas on protecting your keywords, but evidently you've shown that my technique for getting them from competitors works ;). Happy to hear you enjoyed the folder lock thing. I thought it was a clever trick my friend showed me.

    Comment by Gabriel Goldenberg - May 7, 2008 @ 1:34pm
  10. Very interesting, but not too worried. If I check Google Analytics on one site it shows 17,604 key phrases in the last month. Only those at the top of the list would show up in sitemaps. Only about the top ten would show in remote anchor links. And those at the bottom of the list that don't show are the ones that convert.

    Comment by Stephan Miller - May 7, 2008 @ 2:31pm
  11. If you want to see a cooler pay tool which has a free trial check out Adgooroo it will tell you both organic and PPC terms. And how your market penetration is compared to your competitors. Little bit pricey but if your spending a lot on SEO, or PPC then this may save you some money (their advice is very good too). :)

    Comment by Daryl from Quenet SEO - May 10, 2008 @ 1:41am
  12. Stephan, you have an excellent point about this being largely limited to the shorttail. That said, the longtail tends to be composed of the shorttail plus modifiers, so this is still a valid starting point. Furthermore, I've done analytics on sites earning 5 figures/month from single word searches... Daryl, thanks for sharing that. FYI, Spyfu does that and I believe Compete does too.

    Comment by Gabriel Goldenberg - May 11, 2008 @ 1:13pm
  13. Nice work, this article made WebProNews too :)

    Comment by Linda Bustos - May 12, 2008 @ 7:24pm
  14. Hey Gab! Thanks for the link love sir. Check your email, as I'm looking for your MSN/gtalk id -- Dev

    Comment by Dev Basu - May 15, 2008 @ 10:08pm
  15. You can also use Wordze's competitor keywords tool. http://www.wordze.com/competitorkeywords.php And follow it up by using this tool of theirs: http://www.wordze.com/relatedkeywords.php But there always seems to be debate over how accurate wordze is compared to other keyword research tools for some reason.

    Comment by Will - May 22, 2008 @ 1:10am
  16. uh oh, my competition better watch out!

    Comment by failureblogger - May 25, 2008 @ 5:55pm
  17. There are many tools out there which profess to be the end all in keyword research. If chasing the competition was all you had in mind, this is fine. The point is to lead and formulate new thoughts and ideas while embracing the old. Why chase when you can lead.

    Comment by Dave - June 23, 2009 @ 7:56am
  18. Nice tips Gab.I've never tried it before and I never knew that you can protect your keywords.I thought hiding keywords is against the google guidelines.BTW,thanks I found these topic.

    Comment by Andre Colt - October 12, 2009 @ 10:29pm
  19. Agree With u Bro Very Useful For Finding Competitor Step by Step Explanation is awesome ...

    Comment by vikram - March 8, 2012 @ 12:23am
  20. yeah its really difficult to prottect your own keywords when at the same time you have to get your competitors KW :)

    Comment by arun - March 16, 2012 @ 4:39am
  21. Good advice. Is there any plugin for wp that could help hide my sitemap or backlinks? I would be ready to pay for it.

    Comment by Raimond - June 1, 2012 @ 4:55pm
  22. Thanks for this article! Very useful information you have shared. I have heard about the other. Try The Google Wonder Wheel and WordStream offers highly sophisticated grouping tools which facilitate the grouping of thousands of keywords quickly and efficiently. And here are some other useful tools, check it out. http://www.wordstream.com http://www.googlewonderwheel.com http://ubersuggest.org Thank you Gabriel! Margus Internetiturundus

    Comment by Internetiturundus - August 27, 2012 @ 11:02am
  23. I use a different technique but it also has some limitations. Generally what I do is to go to the source code and look at the meta. I always like using Firefox. What I do is to open the website using Firefox. I click on "Tools". In the dropdown menu, select "Web developer" and click on it. In the dropdown menu, select "Page Source". You will be able to view the source code. Look for the keyword meta. You will be able to see the keywords used for that particular page. If it's the homepage, you will be able to know the main keywords of the site. Does that work for you?

    Comment by Charles Kiyimba - November 21, 2012 @ 6:08pm

Leave a Reply