Jimmy writes,

“I saw someone write to be careful or use the ilb in moderation to avoid getting penalized or something like that.

What are your thoughts on best practices?”

I think penalization due to overuse of my WordPress plugin, Internal Link Building (ILB) is a mis-expression of a user design concern.

The consequences of overusing ILB is just that your audience may feel a bit overwhelmed if the same words are linked 5 times in one post. Bing and Google will not penalize you for that.

That said, you can have multiple different words linked in a single post.

Imagine you write about laptops. Your latest post is about a new laptop bag you bought.

‘Laptop accessories’ is mentioned and links to that category on your blog.
‘IBM laptop’ links to your review post
‘TigerDirect’ is an affiliate link
‘Add my rss feed to your reader‘ links to your feed.

Bang – 4 links, 1 post, and it’s just fine.

If you thought this SEO FAQ for using my internal link building plugin was useful, add my rss feed to your reader.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

The situation: Your competitors have inbound links that are broken because of typos, changes in URL structures etc.

The common link building commentary: Most SEOs who’ve been around the link building block will tell you that it’s an opportunity to ‘build links’ for free – to pick low hanging fruit. Just drop the site owner a little email and voila – good as new. More juice for you!

The link-arms-race commentary given by sneaky [and helpful!] SEOs:

“Dear [webmaster],

My name is John Doe and I’m writing because I ran into a few bugs while browsing your site. Namely, there are broken links.

It’s a shame that your otherwise use[less/full] list of sites in your niche should be hampered by broken links and that they’d slow your visitors [Googlebot, and myself my broken link checker] down. So I thought I’d point out those broken links I found for your [annihilation/removal/hexic cleansing].

[Insert list of broken links, first or second amongst which are the competition's.]”

For bonus points:

 Use Virante’s clever PageRank Recovery tool – based off the SEOmoz API – to find all the pages/links on your competitors’ site that are problematic like this. Hat tip to Rand’s post on 30 SEO problems and tools to solve ‘em post.

This reminds me of similar tactics I came up with over 2 years ago in response to the common question, ”Should I buy links if my competitors buy links?”

If you liked this post on link building and such competitive sneakery, add my rss feed to your reader for more!

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

An increasingly popular question in attribution management and in advertising measurement is what effect ads – that were viewed but not clicked on – have. These ads are known in the lingo as ‘View-Thrus’ or ‘View-Throughs.’ The related question people usually have is what percentage of credit to give view-thrus for conversions that came via multiple touchpoints (e.g. the customer saw and/or clicked several ads, before buying).

I’ve come up with an idea for an experiment to measure the effect of such “view-thru” ads on conversion, as well as on acquisition costs. (An acquisition cost is the cost-per-action of acquiring the lead/customer.)

1. First, get a control group and an experimental group.

2. Second, instruct the groups to read and browse a few websites you control, and provide you with feedback on some [non-ad-related] issues with the sites.

The control group will view ads unrelated to what they’re about to search for. The experimental group will view ads by companies also advertising on the terms they’re about to look up.

Why are they browsing around and giving feedback?

The point of asking the groups to read and browse a few websites is twofold. One purpose – for the experimental group alone – is for them to ‘see’ the ads – defined here as causing an impression to be registered by loading the ad along with the rest of the page.

The second purpose – the request for feedback – is to mislead all the users about what the purpose of the experiment is. You don’t want to accidentally induce either group to to look more/less closely at the ads or otherwise behave strangely, but just browse around normally. Scientists have a name for this effect, but I forget what it’s called

At this point, the only difference between the groups should be the display ads they’ve viewed. So any differences between the groups subsequently should be attributable to the ads they’ve seen (even and especially if they haven’t clicked them).

Next, have the groups search for a few specific items and click whatever suits their fancy. The former searches are again to mislead the test subjects about the purpose of the experiment. The latter searches will be about products carried by companies whose display ads they just viewed.

Results and Interpretation

Q1: Is there a difference in conversion between the groups? That difference is attributable to the view-thrus. For attribution-management purposes, you can credit the display ads with the difference in percentage.

So if you convert the control group at 10% and the experimental group at 15%, then display gets 1/3 or 33% credit for the experimental group’s conversions.

(Not ½ or 50% because the denominator is 15, not 10. It’s 33% of 15%, or 1/3 of 15%. You already had 10% conversion rate (CR), so saying display deserves 50% credit would mean that display drove 7.5% conversion rate and the other marketing drove 7.5%. This is false since we know the other marketing, without display, converted at 10% on the control group.)

Note: This higher conversion rate again lowers your cost-per-action, reducing your costs for PPC.

Q2: Is there a difference in CTR between the groups? Supposing view-thrus raise CTR, the higher CTR should lift QS and lower CPC, effectively earning a lower cost-per-action.

So as a fun side-effect of such an experiment, you might find yourself with cost savings in your search budget from both the higher conversion rate and the lower CPC. (You might also find the opposite, such that display ads lowered your CTR and CR, in which case you need to ask whether the branding is worth it for the lower immediate cash flow.) But they came at the cost of a display ad campaign.

Next Steps After Attributing Credit

What should you do, then?

You need to see if the PPC savings with the experimental group were either equal to or greater than the display spend. If you saved on PPC more than you spent on display, you should obviously continue with display. If you saved an equal amount, you probably should still continue with display anyways because you also get a branding benefit.

If you saved less than you spent on display, it’s a question of degree. Are you willing to incur the additional expense for branding benefits? Is the branding a shield for shady SEO tactics? How much higher of a conversion rate did you get?

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

There have been lots of innovations in technology, in the algorithms, in products and search engines (hi Bing!). I’m curious to know what you feel has had the greatest impact on SEO. Feel free to name multiple things, and share practical tips!

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

Is PPC necessary? Why is PPC necessary or not? If so, when do I need to do it?

These are questions I recently got in fielding a proposal for search marketing services. For others who might have the same question, here’s the answer, and an explanation.

PPC is necessary when you’re just entering a market.

At that point, you have no idea what keywords will or won’t convert. By buying ads on those keywords, you can tell what does and doesn’t convert. This is essential to keyword research; you can’t perform SEO that delivers a positive ROI without knowing whether you can make money with traffic from a particular keyword.

Additionally, pay per click ads are extremely valuable to compare the value of keywords before you start your SEO campaign. When you’re just entering a market, you may know that two keywords you’re targeting are commercial, but not which will deliver higher conversion rates, average order sizes, or bottom line value per visitor. PPC can tell you that in a few weeks, whereas SEO would take several months.

Arguably, if you need to compare the relative value of keywords that you know will convert, you can skip PPC. If you’re going to make some money, and that’s enough for you, then you can go straight to SEO.

But if you want to prioritize more lucrative keywords to make the most money – which you MUST do if you’re in a competitive market – then PPC will be necessary.

Why is testing with PPC a MUST for competitive markets?

It’s a must because competitive markets are driven by ROI. Testing keywords, copy and landing paths is the way to generate a higher value per visitor.

In competitive markets, most players are using such testing to maximize their value per visitor and thus their ROI. This means that they can afford to spend more to attract those visitors in the first place.

It’s like a lemonade stand and a Ferrari dealership competing to buy ads in the local paper. Guess who’s going to get the ad inventory? The Ferrari dealership will get to buy the ad space, because it makes more money from the ads, so it can spend more on the ads.

Well, what about if two Ferrari dealerships are competing to buy the same ad space? In that case, it still boils down to whoever can make the most money from the ad. It comes down to better ad design, having salesmen who close a greater percentage of sales, selling more expensive Ferraris or leases etc.

The same thing happens online. Whoever makes more from a given piece of ad space will buy it instead of their competitors. If you content yourself with whatever ROI you can get, competitors who are more serious about maximizing their ROI will eventually price you out of the market. Over time, this means you won’t get any traffic at all.

This is equally true of SEO. If you can only afford some crummy directory submission service, and I can afford advanced link building tactics like linkbait, guess who’s going to rank first?

My backlink research (see also backlink research part 2) shows that in most commercial markets (not just the most competitive ones), you need to build links constantly – just to keep up!  Let alone surpassing others in the rankings.

Testing with PPC makes your business more efficient and grows profit margins. That means you can to spend more on link building and have a chance to succeed with SEO.

When is PPC not at all necessary?

If you’ve already worked in a market and know what keywords convert and what their relative value is, you can skip buying PPC traffic.

Well, kinda. While PPC is great for getting data on the value of a given keyword, it’s also valuable for testing ad copy and landing paths. And if you’re in a competitive market, your competitors are buying PPC traffic to maximize their value per visitor.

And furthermore, the organic search results get more competitive daily. So if you want to be able to afford the SEO services it takes to rank, you need to test with PPC.

To summarize:

• PPC is necessary when you’re new to a market, because PPC lets you test keywords out. This tells you which keywords are worth more money.
• PPC is also valuable to maximize your value per visitor, by testing different messages and landing paths to find out what ads and versions of the website make the most money.
• Competitive markets require PPC. Otherwise, you’ll get priced out of buying any ads or buying seo services to get organic traffic.
• If you already know what keywords are most valuable, you have a lesser need for PPC.
• You probably still need it to be able to test and maximize your profitability, however.
• Update: Josh Sciarrino, of Tampa internet marketing company Refuge Design, highlights that PPC can also be more valuable when bidding on brand, by driving additional marginal sales (be prepared for some cannibalization of your SEO-driven brand sales.

p.s. If you found this interesting, you may want to check out my buddies at Wordstream, who offer keyword research and organization software.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

(To my advanced readers: You know most of what’s in this post. This is to help out the readers newer to SEO, using a question-and-answer style to address frequently asked questions (FAQ).

If all goes well, there should be some feature length, deeply researched stuff for you SEO pros later this week.)

Question: Does it matter for my site’s SEO in Google or other search engines if I use www in the domain name? For example, do I need to use http://www.example.com or can i use http://example.com ?

Answer: It doesn’t matter at all to your SEO in Google, Bing or Yahoo whether you use www in your domain name or not.

Search engines most likely don’t consider that as a ranking factor, because it has no value in improving the relevance of their search results or improving the user experience. There is one exceptional point, however. You don’t want to use both a www version of the site and a non-www version.

If you do use both www and non-www, then you’ll have duplicate content problems, also known as “canonicalization” problems.

(The word “canon” is used in religious terms to mean the “official set of holy writings.” That’s been adapted to other fields. So you can have the canonical set of Harry Potter books, which are the official books, and you can have the “canonical” version of a web page, which is the official one you want search engines to rank in their search results.)

What I mean is that if you have two identical versions of a page, one found at http://www.example.com and another found at http://example.com, and both show up fine to search engine spiders, you will have duplicate content.

This is problematic because a) It confuses search engines as to what is the canonical, or official, page. b) You will sometimes get links to one version and sometimes to another, such that you compete with yourself.

How do you do SEO successfully then, if you currently have such www vs non-www duplicate content problems?

You have two options.

1) Redirect either version to the other one using “301 permanent” redirects. There are numerous tutorials on this online, for DIY people, but you can also get a decent freelance programmer to do it for you. Personally, I get my hosting from Dreamhost and they give you an option to 301 redirect one version to the other.

I have a $50 off discount code at Dreamhost off any annual hosting package (or a $25 off discount on a monthly hosting package) if you need to set up a new site: Just enter GabSavesYou50 when you sign up for Dreamhost hosting.

(Full disclosure: It’s an affiliate link, but I’d endorse Dreamhost personally anyways, as I have in the past. Their tech support is super responsive and they have numerous tools to make things very easy, like one-click WordPress installs.)

2) Use the “canonical” tag to indicate to search engines which version is the official one. For those of you using WordPress, Yoast has a plugin to automate some canonicalization, but I’m not sure it’s applicable to this situation.

(3) Update: Malcolm Coles has a related article with examples of www vs non www that’s a valuable read on this topic.

If you liked this post, you might want to get my latest posts sent to you regularly, for free, by rss or by email.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

… And Based On That Projected ROI, Get Management Buy-In, Set Priorities and Spend Time Wisely.

Note: If you like this post, please link to this page from your blog or share it on StumbleUpon by clicking this button:

Here’s how to project the ROI you can expect on SEO. This way, when your boss asks you to predict next quarter’s SEO numbers to help him with his forecast to analysts, you predictive task will be easy. Better yet, the ROI forecast you make will be defendable.

The detailed explanation on making your forecasts follows after the flow chart.

1) Pick some short-tail or mid-tail keywords. There’s no point picking long tail keywords, because the ROI on them individually is small in absolute terms.

2) Check the keyword for commercial intent.

- Microsoft offers a keyword commercial intent estimator tool. Caveat emptor: The tool’s estimates can vary significantly by day, so just take it as another data point, another road sign along the way.

- Correlate that by asking people to do word associations with the given keyword. You can use Amazon’s Mechanical Turk for that. The associations will show the intent behind a given search.

- Compare the Cost-Per-Click (CPC) estimates Google provides you with for each keyword (set on exact match). The higher the CPC, the likelier there is to be commercial value.

Note: For CPCs under $1, this isn’t very meaningful.

3) Find search volume for your target keywords.

Use Google’s KW tool on exact match to find the exact volumes on target keywords.

It’s crucial to use exact match, and not broad match data, because rankings will result in getting you traffic for your exact match keywords and maybe a few variants. It’s best to be conservative.

3.5) Don’t just accept those search volume projections at face value. Double check.

- Run a PPC campaign.

- If you’re not going to run a PPC campaign (eg if the CPCs are too high), you MUST at least correlate the data to other people’s numbers:

• Use Wikirank, if you see Wikipedia in the top 10.

• Compare with Aaron’s tool.

• Ask friends who’ve been in the same vertical about the search volume. Apparently the accuracy of Google’s numbers varies by vertical, so friends who’ve been there can provide valuable insights. (It’s not too difficult to find such friends once you’ve been in this business for a few years.)

• Also, check against competitors’ keywords. Are you on the right track or really, really clueless?

4) Find out whether you can rank. Scope out the competition:

• Look for backlink numbers and quality,

• Consider backlink growth rates,

• Calculate backlinks/referring domains to find out if there are big brands you need to be aware of,

• Look at their anchor text,

• Consider their own domain name – an indicator of anchor text past, present and future, as well as financial backing for marketing.

5) Estimate your costs including domain, hosting, site design + development, links, man hours.

Note: As pointed out by professional search marketer Simon Serrano in the comments below, certain costs are fixed regardless of whether you do SEO or not. So if you’re doing this with an existing site, then you can exclude those costs. But you’ll probably compensate with the man hours etc.

My perspective in suggesting that the domain and hosting costs be included was that of someone starting a new site from scratch, with the mentality, “I’ll rank and sell to search traffic.” For them, the exact match domain might be a big cost that wouldn’t be incurred otherwise, for example.

6) Estimate the traffic attainable from various positions. See Aaron’s big resource page on the topic and scroll down about 1/3 of the way for the breakdown of Click-Through-Rate (CTR) percentage by position.

Hat tip to my friend Henry Shih of online jeweler Ice.com for coming up with the idea to “make a range of projections, from conservative to optimistic” last summer. Todd Friesen has also suggested this.

Update: Given Universal search results with maps, news, videos, images etc., you may want to review these results downwards to be conservative. As Simon points out in the comments, the CTR data has changed significantly since the time AOL leaked its numbers.

7) Calculate your potential revenue.

If you’re an affiliate: i) Multiply those attainable traffic numbers by your CTR. ii) Multiply again by the merchant’s conversion rate. iii) Finally, multiply by your average commission. Take multiple values for both your own CTR and the merchant’s conversion rate, so you have a range of estimates from conservative to optimistic.

To simplify, if you know what your earnings per click will be (eg recently work in the niche), you can multiply traffic by EPC.

7b) If you’re an adcents publisher: Multiply that by your CTR and EPC if you’re selling CPC ads.

7c) If you’re a retailer / lead buyer: Multiply that by your conversion rate and then by your average if you’re the merchant.

Finally, divide your potential revenue by costs. This gives you your SEO ROI – the return on investment for each dollar you put into SEO.

I do this in a spreadsheet for convenience. If you add my rss feed to your reader, you can download a copy of the spreadsheet for your own use.

This has literally saved me thousands of dollars and countless hours by helping me avoid niches that offered a lower ROI for the same time and money investment…

p.s. If you want to know how to prove SEO ROI on past activities, that’s easy. Open your web analytics, and see how many organic visitors on non-brand keywords turned into leads or sales. Alternately, show your rankings increase if you’re not yet in the top 10, and thus not driving traffic just yet.

Found this post useful? Add my feed to your reader for more advanced SEO tips for inhouse SEOs, consultantts and independent webmasters and affiliates! If you want a price estimate for SEO services, you can get one quickly here.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

A visitor dropped by my site asking whether Google cared about a link’s styling. I got in touch with Matt Cutts, and he was nice enough to take a few minutes to share an answer. Here’s what Matt said:

“This is something that can go both ways. On a graphics-heavy site where it was the design idiom to go non-underlined until you hovered over a link, I wouldn’t foresee that being an issue.

If we thought that people would doing it in a deceptive way though, that could be higher risk. I documented a good example of this here: http://www.mattcutts.com/blog/hidden-links/ “

On that post, Matt addresses spammy links. In particular, he highlights the use of javascript to hide the fact that links are links. The javascript prevents the cursor from changing to the “this-is-clickable-hand-icon.” Simultaneously, the link had CSS applied to make it look like ordinary text…

So if it’s just a style element, Google probably doesn’t care. If it’s being used to deceive users, particularly in conjunction with other code that will make it impossible to discern a link short of viewing the source code, then you’re taking risks.

As an aside, it’s better usability (see Conversion Dr’s test),  to make links underlined, since that is what users expect. If you MUST go non-standard, tedster and other WMW users have some worthwhile comments. And the dean of usability himself, Jakob Nielsen, has this Alertbox with particular detail for intranets etc.

P.s. It’s worth repeating that at the end of the day, the ranking paradigm at Google is user experience, though brand building is also key, so why not check out my brand building guide for SEOs and Internet Marketers?

If you liked this seo faq, why not add my rss feed to your reader? Or if you want to read more right now, check out this related tip to sexify your SEO and please both SEOs/web designers.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

A reader from Singapour asked this to Google and then came to my blog: “Why is search tied to ROI?” ROI means return on investment, an essential business “metric,” or “measurement standard.” The answer to why search marketing is tied to ROI is in the full post (click read more for access).

Good question. The answer is that people want measurable results so that they can try to improve (aka testing) and then see if their effort was succesful. If you can’t measure, you’re wasting your money investing into search marketing, because you’ll

a – Never know if search is working or not.
b – Never know what technique, keywords, or campaign is providing a positive ROI and what is providing a negative ROI.
c – Never be able to improve your efforts/results nor duplicate them. My dad did a very succesful direct mail campaign, had no offline analytics to measure why it worked and has since been unable to replicate the same level of success.

As a starting point, you should try out Conversion Ruler, Sitemeter and Google Analytics – they’re all cheap or free web analytics solutions that can get you started measuring your efforts.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

You often hear best practices saying that Google won’t index your pages if they force Googlebot to take a sessionId. Is that really true?

How does Googlebot treat session IDs?

http://www.google.com/search?q=inurl%3A%22sessionID%3D%22

Found while looking for folks using Blue Martini, which seems to output some nasty long URLs featuring BM user IDs and “J sessionIDs”: http://www.google.com/search?q=inurl%3A%22bmuid%3D%22

Like this post on session IDs? Get my RSS feed.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

“What does Google want” is a common question that many pretentious SEOs claim to know the answer to. I’m about to join their number. Google has moved beyond measuring SERP quality based on relevance and are now aiming to provide the best user experience possible.

Other titles I was considering for this post were:

Why Matt Cutts’ “Make Content For Users” Was Very Insightful

How Googlers Measure SERP Quality – Relevance Is No Longer King

Sorry Rand, But The Googlers Were Very Expressive, IMHO.

They are aiming not just for relevance, but overall positive user experience.

That’s what’s behind labelling of cracked sites in SERPs. That’s the reason for Universal search. That’s why AdWords integrates with GA, and GA with Feedburner. That’s why Quality Score counts loading times.
That’s probably what’s behind Knols (remember, Wiki + AdSense is good user experience .) For the inspiration to this post, lookie : what google wants/researching the territory.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

The answer came to me while reading up on advertising. Studies show it takes a certain frequency – most people place it around 7 times – for an ad and its message to be remembered. It would obviously be silly to just credit the last impression for finally getting the target consumer to get the advertiser’s point when the other 6 clearly were part of the process. Yet that’s a question many pro marketers have!

“Do we credit the first click or the last click for this conversion?”

Answer: You credit them both and the ones in the middle too. They all played a role. If it turns out your cost-per-action (CPA) is higher than projected now, that’s tough luck – factor it into your measurements and budgeting. You can’t count a single 50 cent click for the conversion when there were 3 other dollars that also made an impact on this new client…

Also, if one or more of those impressions came from the organic side of things, you can either measure the value at PPC cost. E.g. if you’d have to pay a dollar to buy the click, that’s how it factors into the CPA calculation. An alternative is to measure the value of labour/other expenses to rank divided by total traffic pulled on the keyword to get the cost. Obviously this is more complicated, so PPC replacement might be simpler to measure.

This answer to the source attribution question also affects how you structure affiliate commissions and payouts. Out of the 7 repeat visits prior to purchase, how many did the affiliate send? 1? 2? 4? 7? Sure you don’t want to tick off affiliates, but you need to credit your own inhous work too. Also, keep in mind some people will purchase on visit 1 and others will purchase on visits 3, 5, 8 or 11. You might need to get more sophisticated with your analytics…

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

Update: Check out this “How to forecast SEO ROI” article if calculating the ROI on SEO is the info you want.

What follows is an editorial / research article showing that SEO is valuable – but not explaining how to calculate it.

I was asked what the ROI on SEO is a few times at a recent business event, and decided that it was about time someone spoke up for us organic search marketing experts. The sad truth is that we SEO Experts are grossly underpaid! Let’s look at some stats (or damned lies, if you prefer).

According to a SEMPO report, the North American Search Marketing industry grew to $12 Billion in 2007. (Worldwide numbers are unavailable, but presumably much larger.)

According to the same report, 88% of search marketing dollars were spent on PPC while only 10.5% of that went to SEO. Various miscellaneous categories, including an amorphous “technology investment” (Analytics? Content management systems? Web design? Servers? …) capture the remaining handful of percentage points.

An older study from 2004, by Internet Retailer, showed respondents saying that the conversion rate of organic traffic (i.e. the traffic gained from SEO) was greater than that of PPC.

“Nearly one-half—46.1%—maintain that natural search delivers better sales conversions vs. 37.3% who cite pay-per-click as a better conversion generator and 16.6% who say paid and natural search perform equally as well.”

Conversion rate, for those of you who may be newer to the search marketing business, is the percentage of visitors who buy. If 3 out of 100 visitors buy your blue rattle, your conversion rate is 3%. So what the majority of the retailers above are saying, in other words, is that the ROI on SEO is better than that on PPC. This old SEO Chat case study also shares a higher conversion rate on organic results.

But wait, there’s more. Consider the fact that a majority of clicks on a page of search results go to the organic listings. I’ve just spent an hour and can’t figure out the keywords to unlock the studies to “prove” this, but since it’s so commonly known, I’ll ask you to take my word for now. If you do have a study to share on the CTR for SEO vs PPC, I’d be happy to see it.

Update: In the comments, Antonio of Marketing de Busca shared the following great post with data straight from the horse’s mouth: Avinash Kaushik (consultant at Google) cites 86% of clicks as going to the organic results and 14% going to the sponsored listings.

So, to summarize, here’s the situation in the search marketing industry.

• A landslide majority of the money spent on search marketing
• Is going to attract a minority of the search engine traffic,
• Which paid traffic provides a lower ROI than SEO.

Let’s be clear. PPC is a great way to build business and gain sales, and it should be a key part of any balanced search campaign. There’s no doubt about it; only doing SEO can have you waste months optimizing for keywords that convert less well than others you could have targeted!

But there’s also no doubt that there’s a huge ROI opportunity in SEO because such a relatively small percentage of search budgets is going to deliver organic results, not to mention that those results deliver a better ROI than PPC results. I look forward to hearing and responding to your comments, and suggest that if you liked this post, you subscribe to my blog’s RSS feed. You may also care to read this related guest post I wrote entitled, “What Every SEO Needs to Know about Branded Search ROI” and this on the three types of SEO consultant: Good, Rad, and Sexy.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

I’ve been asked the question recently in connection to business blogs: Should I blog on the company’s official site/domain name, or should I blog on on a fresh domain name? Each approach has its advantages, but with current search engine algorithms, my advice is to have the blog on the company’s main domain.

Blogging on the main corporate domain

Pros include:

• Traffic can transit over to the blog from the main company site and quickly build up the blog’s authority, share its posts and ideas, and connect to the company through its blogger(s). This assumes you don’t ignore your blog, like Facebook.
• Links go to the main domain that you’re trying to rank, which is key given the importance Google places on authoritative domains.
• If your blog links to inner FAQ pages or other things on the same site, the look and feel will be the same; the traffic is less likely to bounce upon arrival to the pages you linked to.
• Centralization if you have multiple company blogs (they can all go on one domain), as opposed to spreading them out over a disparate number of domains.

Cons include:

• You could get an extra source of “external links” by putting the new blog on a separate domain (and hosting on a different C block of IPs, at least for Google’s algorithms; hat tip Wiep’s link roundup).
• People may be reluctant to link to a “corporate” blog.
• If the bloggers become popular enough, they can just leave and blog independently.

Blogging on a new domain

Pros include:

• An extra source of links to your new site. I say source and note just ‘more links’ because my own experience with my old Montreal SEO site and this one is that after the first couple of links, additional ones aren’t very helpful, unless they’re deeplinks.
• It’ll be easier to attract links to the blog, though if the blog itself doesn’t convert visitors (either from the links or from the search engines), the value of this may be questionable.
• You might have a more relaxed graphic design, like being in the company lounge as opposed to the company press conference. This could, in turn, attract links and positive attention from CSS galleries and help enhance your company’s “cool factor” with web designers.

Cons include:

• More traffic will bounce when it hits the company’s official domain.
• Traffic won’t transit over from the main site as easily (plus you’ll have more bounces there too), so you won’t have as easy a time spreading the word at first.
• Links don’t go to the corporate domain, so you won’t build up its authority as easily.
• Doesn’t scale as well, inherently.

At the end of the day, the domain authority/strength issue is the big differentiator where I’m concerned. If you check out Wiep’s piece on how he had Matt Cutts’ site (about search marketing) rank for one of the most competitive phrases around, you’ll understand what I mean. Matt’s page had a single inbound link whereas players in that space usually have a pyramid of tens of thousands of links backing them up; the ranking was mostly due to his site itself having so many links and being such a trust authority.

Update: Romela of Business Solutions has a good addition in the comments. So does Dave, of Zoekmachine Optimalisatie SEO.

So keep those corporate / company / business blogs on the main company website. If you liked this, subcribe to my RSS feed.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

Social media is one of the most difficult things to justify in terms of ROI because current analytics aren’t well suited to measure its data. Here’s my proposal for social media analytics and tracking. This is an approach to use as a foundation for creating social media analytics tools, not a tool.

If you want tools, then check out the tools listed at the end of this article. Unfortunately, the best they can currently do is just measure brand mentions and track them; that only solves part of the problem as we’ll see.
(Soon to be wed by Simoty77/SLloydBottom)

Social media is a suite of tools for … socializing.

Therefore, the best measure of success in social media is how many relationships you have and how strong your relationships are.

Personally, I really enjoy linking to other people and/or submitting their stuff to Sphinn because I know it strengthens my relationships with them.

To reprise Sean Covey Jr.’s metaphor from his highly succesful 7 Habits of Highly Effective Teens (reprised from his dad’s 7 Habits of Highly Effective people?), you’re putting a deposit into your relationship bank account. The more deposits, the richer you are.

The reason regular analytics aren’t suited to measuring this is because they collect clickstream data (word kudos to Avinash Kaushik). Whether we’re talking about logs or about javascript tags or sniffer packets, the data is what occurs in your visitors’ browsers. Social media analytics need to track what goes on in your acquaintances and friends’ minds. (Like the thought police, only different . )

That’s only a slight exaggeration. What I’m getting at is that you want to measure your network of friends and contacts. Since I’ve been getting more active and having more success on StumbleUpon, I know that my network is growing in size, and as I submit more content and thumb more up, in strength as well. Is your network a sturdy jungle gym or a flimsy spiderweb? (Jungle Gym by bazzmc.) If you’re going to measure your social media results for now, the best you can do is sit down with a spreadsheet, write up your friends’ names and what you’ve done for them recently. Have you:

• Connected them with useful contacts?
• Sent business their way?
• Linked to them?
• Interviewed them for something?
• Answered some questions they had?
• Gotten them a ahead in some way, shape or form?

I emphasize the recency bit. To quote the Japanese , “A kind word can warm three cold winter months.” (Which means that if you’ll see a higher ROI in Canada.) More seriously, this will help you assess where you stand, because people forget favours quickly remember injuries extensively. I still remember which kids stole my hockey cards in grade school, and that was over a decade ago! Ask me who gave me a compliment, and I’d be hardpressed to tell. (Though that may be because I was a bit of a loner back then.)

Another advantage of this approach to tracking is that it’s more actionable than clickstream data. Compare “I haven’t been in touch with Sylvain in a while…” vs “We got 348 visitors today.”

On that same note, I’d like to highlight that I’m far from being the only one discussing this issue. Get some further reading from the following folks, who address social media, ROI and measurement, though none of them quite in these words, afaik.

Note: While Ann and Maki are pretty close to my thinking, I believe this post adds to the discussion (especially in terms of specifics, and pointing out the obvious which for some reason hasn’t been addressed yet) rather than just rehashing it.

1. Maki – social-media-networking-and-roi/
2. MindValley Labs – whats-a-friend-worth-to-you/394/
3. Ignite (ft. Brian) – social-media-metrics-coming-to-an-algorithm-near-you-part-1/
4. Huomah – social-media-marketing-is-it-for-you.html
5. Huomah – the-Value-of-Social-Media-Marketing-Part-II.html
6. SEOmoz in part – whiteboard-friday-tracking-nontraditional-conversions
7. Annie at SEOmoz – creative-rss-button-could-it-work
8. Annie @ home – http://www.seosmarty.com/social-media-marketing-think-long-term/
9. Ann’s touched on it elsewhere (or was that just PMs?) but I can’t find it. You get the point though, the gal’s smart.
10. Yours truly – seo-roi-tops-200-subscribers-case-study-on-feed-analytics-and-poll/ (update: tops 300 )

To conclude, consider this illustration of the ROI you can get from social media.

Guy Kawasaki, when explaining how he was able to launch a site that got 200,000 pageviews its first day, said, “I spent 24 years schmoozing and paying it forward.”

And that, ladies and gents, was my proposal for social media analytics and tracking. Like this person said: Measure relationships, their strength and their growth.

On a related note, you may care to read this post on the measures and value of attention equity: How do you measure attention and what is it worth to you? If you think you’re likely to visit again why not just add my rss feed to your rss-feedreader ?

Existing social media analytics tools are as follows. Note that they’re mostly quantitative, rather than qualitative.

To analyze Digg, Mashable lists 5 tools.

What are people talking about? Try Facebook Lexicon.

Marty has a great item on building a reputation monitoring dashboard. He also has buzz pocket mining tools for those interested in mining social media for keywords.

Some other miscellaneous ones include Keotag – for tracking tag use ; Boardtracker – for forum conversations in particular ; Google Blogsearch, which lets you track blogs that mentions your desired keywods (e.g. Brands, products etc.); Copernic.com – for changes on a site ; and Radian 6, which Ben has reviewed.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

I was emailing a prospect recently who mentioned that a competing firm had proposed doing A/B multivariate testing. If you’re familiar with the jargon of testing different ads/landing pages, you would know that A/B testing is different from multivariate testing. I can’t blame the prospect or my competition however, because ours is an industry enamoured with jargon and it sometimes gets me confused too! In any case, let’s see what A/B testing and multivariate testing really mean, and the implications for your test results.

(Bucket testing image by mil8.)

A/B testing is defined as testing different values for the same variable. For example, suppose you have three landing pages, called A, B, and C. You can carry out A/B testing by changing the titles on each landing page. The variable there would be the title, and each the A’s title, B’s title and C’s title would each be a value for that variable.

It isn’t about having only two versions and comparing one to the other. You can do A/B/C testing or even go super deep and do A/B/[...]/Z testing! A/B testing is also sometimes referred to as split testing or A/B split testing because you split your traffic (or your “sample,” if you’re testing something offline, like in-store displays)

Multivariate testing is defined as testing multiple values for multiple variables, simultaneously (hence the name “multivariate”). Let’s say we’re still working with pages A, B and C. Instead of just testing different titles, we’d also try different colour ‘buy now’ buttons, emphasizing different benefits in the copy and displaying different testimonials.

This is all done at the same time. If you tested each of these variables separately, you’d really just be carrying out A/B testing!

To give a human analogy, A/B testing is like having two brothers growing up side-by-side. They’re raised exactly the same way and are identical in every way, except that they play different sports. You’d be testing for the result of being enrolled in each of those sports and see who got a better job, or earned more money, for example.

Multivariate testing would take those brothers’ children, and have them being raised very differently. The variables there would be the brothers’ different wives, the ways they were being raised, their environments etc. You could still test to see who got better jobs and/or earned more money.

As this example shows, there’s a big difference in what you can learn from the results of these two types of testing.

With the two brothers, we know that since everything else was the same, any difference in job quality or salary is directly attributable to the sports they played. If the soccer-playing one got a better job, then you know that raising a kid to play soccer is better for the child than raising him on say, baseball. (Statistical significance is another issue for another day.)

With the various cousins, we can make educated guesses about what caused the differences in job quality and salary. Perhaps brother A’s wife worked in the recruitment industry and helped her kids more than her sister-in-law could help her kids. Perhaps the neighbourhood where brother B and his wife raised their kids didn’t have as good schools. You can make good guesses about why you got a particular result, but you can’t be certain.

Note: To my more advanced readers, I’m vaguely aware that depending on your sample size and how you play with the numbers, you can get a very high degree of certainty (if not 100%) with multivariate testing. My point above is made at a basic level, and is the main distinction between A/B testing and multivariate testing.

Update 1: Sphinn button is broken, but you can [should? ] sphinn the story here. Also, I forgot to point it out, but if analytics are your bag, check out these 3 new metrics for measuring social media and this post on messing with competitors’ analytics.

Update 2: Tim from Convert Offline gets our first dofollow of the day . He’s a local hound apparently. Me too! Miriam’s going to interview him too, so keep your eyes peeled! (No, that doesn’t mean that you should put a sharp metal implement near your cornea. )

Update 3: More link love abounds! The very succesful Brian Chappell gets his and Hannah Smith gets some for the Gravy Train and her post on rogue Adwords due to personalization.

Please let me know in the comments whether this helped clear up to you what A/B testing is vs multivariate testing. Also, please do comment about how you’re using A/B testing and/or multivariate testing and any case studies/experience you might have in the matter! Testing is a fascinating topic and while I know a little bit about it, I’m sure many of you could teach me more! Good comments get dofollow links, as usual.
Liked the post? Get my rss feed.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

First of all, submit to directories that review submissions before accepting them. The whole premise upon which modern search engines are built is that a link from one website to another is an editorial vote as to the quality of that website.

Aside: Several people who’ve read this have written to ask me questions regarding directories. While I’m glad to lend a hand, you should also consider buying an hour of my time. You’ll get to ask me as many questions as you can fit in to the hour, and I’ll give you the answers you need to improve your SEO.

Directories that manually review submissions before including them are the perfect example of this theory. Human editors ensure that only those sites that meet certain standards are included, and thus vote, vouch, or whatever other term you like, for the website.

Google engineer Matt Cutts recently suggested criteria by which to judge a directory’s quality.

“When considering submitting to a directory, I’d ask questions like:
- Does the directory reject urls? If every url passes a review, the directory gets closer to just a list of links or a free-for-all link site. (http://en.wikipedia.org/wiki/Link_farm)
- What is the quality of urls in the directory? Suppose a site rejects 25% of submissions, but the urls that are accepted/listed are still quite low-quality or spammy. That doesn’t speak well to the quality of the directory.
- If there is a fee, what’s the purpose of the fee? For a high-quality directory, the fee is primarily for the time/effort for someone to do a genuine evaluation of a url or site.
Those are a few factors I’d consider. If you put on your user hat and ask ‘Does this seem like a high-quality directory to me?’ you can usually get a pretty good sense as well, or ask a few friends for their take on a particular directory.”

Second, submit to directories from which you expect to get quality traffic that might buy your products or services. Generally speaking, this means the biggest general directories as well as the smaller industry-specific directories. Some directories are combined with vertical search engines, and these should also be considered. For example, Business.com is both a vertical search engine (i.e. for business websites) and a directory.

Some good directories to submit to are the Yahoo directory (dir.yahoo.com), Business.com, Best of the Web (BOTW.org), and JoeAnt. Depending on the nature of your business, internet yellow pages and online local directories might be a good idea too.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

It is worth submitting your site to some directories, including paid ones. There are two reasons to submit to directories. First, you stand to gain text links that help with your SEO. Second, some directories will also send you traffic.

Aside: Several people who’ve read this have written to ask me questions regarding directories. While I’m glad to lend a hand, you should also consider buying an hour of my time. You’ll get to ask me as many questions as you can fit in to the hour, and I’ll give you the answers you need to improve your SEO.

For example, if you’re unsure of the quality of some directories you’re considering submitting to, we can talk it over and analyze their quality. You also get a transscript of our conversation or mp3 recording so that you can review what we said at a later date and apply the advice when you’re ready. And to top it off, I guarantee your satisfaction. If you aren’t satisfied, just say so and I’ll refund your money, no questions asked. Now, back to our regular programming…

Text links from outside sites indicate to the search engines that your website is a quality one. It’s as if the owner/administrator of the other website says, “I know this site and am vouching for it.” Additionally, the anchor text will help indicate to search engines what the site is about. In other words, the blue underlined words determine what keywords your site (or page being linked to) is relevant for.

Second, if you sign up with well-maintained and professional directories like Yahoo, you’ll find that some visitors will come to your site through those links. It won’t be much, typically, but if you can get some extra traffic and perhaps a few leads or sales, then why complain, right?

For more information, consider reading which directories are worth submitting to.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

A question I’ve been asked a lot recently is “Can SEO be automated? Can I just get some software or hardware that’ll do the job automatically and get my site ranked at the top of the SERPs for my keywords?”

No, you can’t automate it.

First of all, search engine optimization’s foundation and core is about building links. Take the example of Google-bombing, where many people link to a certain webpage with the same anchor text, in a concerted attempt to have that page rank for the words in the anchor text, even though that page is not trying to rank for those keywords. This technique was succesfully used on several occasions and serves to demonstrate the power of link-building.

Now consider how you go about building those links. You contact webmasters and other people with websites and ask them to link to you. It helps a lot to be friendly and give them feedback on their own site, mentioning broken links or forms and other things that could be improved. And to a large extent, you want to go on to develop and maintain relationships with as many of those webmasters as you can, because that’ll get you other links in addition to that first one. Just consider what Marketing Sherpa’s Wisdom 2007 publication quoted me on (look at #27). You need to network to build links.

No matter how advanced spam-email software gets, it won’t network with other webmasters for you. It might let you include the url of the webmaster’s site in your email, but you’re still not getting a link. First of all, because these things show you don’t care about the webmaster you want a favour from. So why should they care about you? The reaction goes something like: “Generic spam email? Delete. ”

So that’s why you’re never getting anything to automate link-building. Right there, you’ve just lost your chance at ranking for anything mildly competitive.

The second reason SEO can’t be automated is because the software that’s around is just plain garbage. I tried using one of these things just to do a site analysis of my own blog. I figured it couldn’t go wrong just reading code and telling me what was done well and what was done wrong. But this totally inept piece of software came back to me and told me it couldn’t find my keywords once on the page (I wish I remembered the name to help you steer clear but this was a looooong time ago – probably a year or more. Edit: I just saw more people asking this question on a forum I read. The software I tried was WebCEO.) . This was on my Montreal SEO blog, which, wouldn’t you know it, ranks for Montreal SEO!

Well, what about keyword research, you say? Keyword research is perhaps a minor exception to the rule here. You’ve got tools that will help you discover new keywords to try and rank for. But these are mostly for preliminary keyword research aimed at search engine marketing, or SEM. The distinction is important because if you just try and rank for the highest (or lowest) volume keyword in your niche, without having run an SEM campaign beforehand, you might end up ranking and waisting your bandwidth on unproductive traffic.

Imagine getting 10,000 visitors a day to your sports equipment website from people who are lazy and hate sports. Not only did you waste your time, effort and money achieving that ranking, but you’ve actually got hosting costs accumulating on top of it all!

Bottom line: SEO can’t be automated.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  

Q: I’ve heard that I should have robots.txt on my website. What is robots.txt and why should I use it on my site? And if I really do need it, how do I go about writing and implementing robots.txt?

A: Robots.txt is a text file that tells search engine spiders (what are search engine spiders?), also known as search engine robots, which parts of your website they can enter and which parts they can’t. You can think of it as a nightclub’s bouncer or doorman, whose job it is to keep certain people (in our case the search engine robots) out of a certain VIP section of the club.

Now you may be thinking, I thought SEO was supposed to get my website fully crawled and indexed by the search engines! Why in the world would I want to keep their robots away from some of my pages? It’s a good question, and the simple answer is that most websites don’t want any of their pages being hidden from the search engine spiders.

The main reason why robots.txt would be used is to keep sensitive information private.

For example, if an ecommerce site uses a database with clients’ personal information stored on it, and this information is accessible through the website, robots.txt would be used to tell search engine spiders not to crawl (and therefore not to index) the access pages. If Yahoo’s Slurp robot collected the site’s clients’ credit card information, then anyone searching Yahoo could access those numbers! To avoid this PR and legal nightmare, robots.txt would be used to hide the database from Yahoo.

Another case for using robots.txt would be the situation where expensive research was being posted online. For example, imagine that pharmaceutical giant Pfizer had its research department using a wiki (a sort of collaborative website to which anyone can easily contribute and post updates) through Pfizer.com to share ideas and discoveries. It could use Robots.txt to make the wiki off-limits to search engine spiders.

Important Note: Password protection, firewalls, encryption and other security devices should be used in conjunction with robots.txt. Search engines are only one of many ways people come to visit a site, and so the security of sensitive data needs to be considered in light of human surfing behaviour, not just robots. Besides which, malicious spammers also use spiders and robots, and the first part of a website they’ll go to is often what’s been marked “Do not enter” in the robots.txt file. They know that’s where the “valuables” are kept.

At the time of this writing, Mcdonalds.com’s robots.txt excluded robots from crawling a directory called tools. If you were to try to visit Mcdonalds.com/tools, you would get an Error: file not found page (both by trying http://mcdonalds.com/tools and http://www.mcdonalds.com/tools ; see discussion below about canonicalization problems). Other companies simply redirect such queries to the homepage. Both approaches are valid, and worth implementing.

Robots.txt is also used to avoid what is known as “canonicalization” problems or having multiple “canonical” URLs. This problem is sometimes referred to incorrectly as a “duplicate content” problem.

Canonicalization problems occur where multiple pages on a website contain the same information. For example, a product page might also have a “Print” version to make it easier to print out the specs and details on the product. This can pose a problem to search engines who then have to figure out which version of the page (i.e. site.com/product-a.html or site.com/product-a-print.html) is the canonical one. Robots.txt would be used to keep the secondary version(s) from being indexed.

However, this solution isn’t ideal, because some people will link to a secondary page. If robots have been told to keep out, then the SEO benefit conferred to the from the inbound link may be lost. A better alternative may be to redirect search engines from all the secondary pages to the primary page, that way all links will certainly be counted. In addition the links will be counted towards the benefit of one specific page (as opposed to two or more), making it easier to achieve high rankings and pull in traffic.

(In case you’re wondering, “canonicalization” and its related words come from the world of religious writing. There, books or theories and other ideas may develop for a certain time until they are “canonized” – declared the best – and may no longer be edited. It is then part of the “Canon” or official doctrine.

As to the problem being referred to as a “duplicate content” issue, this is incorrect because duplicate content refers to a situation where more than two or more websites share the same content. Obviously, robots.txt can’t disallow pages on someone else’s website and is therefore useless in a duplicate content situation. Here the duplicate versions of the content are all on the same website. The difference is like that of a big company’s IT department and marketing department both publishing separate but identical press releases (the canonicalization issue) as opposed to two rival companies both claiming a new discovery as their own.)

How to write robots.txt and implement it on your website

The first thing you need to do is write a robots.txt file (i.e. create one).
Open Notepad (Mac users use a text editor that can output “.txt” format files) by clicking Start -> Accessories -> Notepad.

Now, type (or copy-paste) the following two lines into Notepad.

User-agent:
Disallow:

Next, save your file as “robots.txt”. The .txt extension should be on by default if you’re using Notepad. You just need to save the file as “robots” (without the quotation marks). Be sure to use lower-case, because some technologies may not work well with a “Robots.txt”, for example.

The fourth step is to customize the robots.txt file by writing out which robots you want to exclude (by altering the User-Agent line) and from what files or folders you want to exclude them. Below are some examples from the official robots.txt site. Once you’re done customizing your robots.txt file, you just upload it to the root directory of your website (it won’t work if you upload it anywhere else, such as a subdirectory).

What to put into the robots.txt file

The “/robots.txt” file usually contains a record looking like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded. [Namely the cgi-bin, tmp and~joe directories. A directory in server jargon is a file folder: it's a place where other files are kept and from which they may be accessed. By disallowing an entire directory, you block all the files it contains from search engine robots. These directories should not be confused with yellow-page style directories that list websites, such as Joe Ant.]

Note that you need a separate “Disallow” line for every URL prefix you want to exclude — you cannot say “Disallow: /cgi-bin/ /tmp/”. Also, you may not have blank lines in a record, as they are used to delimit multiple records.

Note also that regular expression are not supported in either the User-agent or Disallow lines. The ‘*’ in the User-agent field is a special value meaning “any robot”. Specifically, you cannot have lines like “Disallow: /tmp/*” or “Disallow: *.gif”.

What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:

To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *
Disallow:

Or create an empty “/robots.txt” file.

To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

To exclude a single robot
User-agent: BadBot
Disallow: /

To allow a single robot
User-agent: WebCrawler
Disallow:

User-agent: *
Disallow: /

To exclude all files except one
This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “docs”, and leave the one file in the level above this directory:
User-agent: *
Disallow: /~joe/docs/

Alternatively you can explicitly disallow all disallowed pages:
User-agent: *
Disallow: /~joe/private.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

Like this post on Robots.txt? Subscribe to my RSS feed then.

Downloads For RSS Subscribers:

• Updated Feb 8, 2010: Internal Link Building, a WordPress plugin. It optimizes your internal navigation so that you can rank like Wikipedia. General info: Check the Internal Link Building update page.
• ROI on SEO Calculator spreadsheet.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data