This is a guest post from Everett Sizemore, who is an eCommerce SEO consultant operating off his 38 acre farm in the Blue Ridge mountains of Virginia. He enjoys gardening, collecting eggs and tackling tough SEO problems.

Want to fight the scrapers outranking you with your own content post-Panda?

Google is losing the war against content scrapers and we’re the ones paying the price. It has gotten even worse since Panda, despite efforts to fix the problem. I know of several dozen websites that are being outranked by scrapers.

While Google might hope we’re going to do their job for them with tags like rel canonical, and rel author, this problem isn’g going away any time soon. You know things are bad when powerhouses like TIME.com are outranked by sites like this.

Fortunately, there are many ways to keep sites from scraping your content, including easy ones like simply asking them to remove it, or bureaucratic messes like filing a DMCA complaint and reporting it to Google. But neither of these options is practical when your’e dealing with more than a few sites or a few dozen pages of content.

In looking for an effective strategy to combat scraper bots you should consider a multi-faceted approach that is scaleable, and then pick up any that get through the cracks using cease and desist letters and reports to Google and/or their host.

The Multi-Faceted Approach To Fighting Scrapers:

#1 – Due Dilligence
First, make sure your message is heard loud and clear. Have your copyright visible in the footer of every page, and include something like the following on your Terms page, as well as in an area commented out within the body copy code:

<!– COPYRIGHT NOTICE: The content of this page is owned by MyWebsite.com and is protected under copyright law. All rights are reserved. Using this content in any way is forbidden, except when permission has been given explicitly in writing from MyWebsite.com. Any use outside the given permissions constitutes copyright violation and will be prosecuted within the full extent of the law. All access to this website is logged. –>

While that probably won’t stop a lot of scrapers, it provides you with the first line of defense and shows that you have done your due diligence from a legal perspective should you have to file a DMCA report.

#2 – Robots.txt
Disallow any bots that you don’t want accessing your site from your robots.txt file. While a lot of scrapers will ignore your robots.txt file (more on that later), many of the popular tools used to scrape websites do respect this file. As an example, you might consider these two lines:
User-agent: Yahoo Pipes 1.0 Disallow: /
User-agent: Yahoo Pipes 2.0 Disallow: /

Disallow a specific file and then record any traffic that accesses the file, which should also have a robots noindex, nofollow tag on it. For example…
Disallow: /do-not-open.html

#2.5 – Bait the Trap
Now link to the disallowed file from within the body area of every page on your site. It can just be an href tag, or you can put the link around a 1 pixel image if you like. All of the “real” search engines will respect your robots.txt file and not visit that URL. If they don’t respect that file then you can bet it’s a rogue bot, even if they are “spoofing” Googlebot’s user-agent.

#3 – Htaccess (Or your Windows server equivalent.)
You should be recording which IP addresses, referrers and user agents access that do-not-open.html page, whether with a script or just by viewing the log files. The next step is to block all of those from accessing your site. You have some options here. For instance, you could deny them access, send them to a custom page (like ScrewYou.html), or get a little tricky and redirect them to their own IP address, which has the benefit of crashing some bots by sending them through an infinite loop. Eventually they’ll get the point and just stop coming to your site.

You can try and get a head-start on the process by blocking the scrapers that are already on blacklists like…
Referrer blacklist by Perishable Press
User Agent blacklist by Perishable Press

For more technical instructions, including the code to use in your htaccess file, see this great post on PerishablePress.com: Eight Ways to Blacklist with Apache’s mod_rewrite

There are probably as many ways to deal with content scrapers as there are ways to scrape content, including several WordPress plugins that mostly just ensure the scrapers give you a link by embedding them into your RSS feeds. What are your favorite methods and why?

[Editor's footnote: Conversely, if you have legitimate uses for scrapers, here's an article on three web screen scrapers for SEO. And if you liked this post, add my rss feed to your reader! ]

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

Fellow Canadian SEO, Melanie Nathan, recently wrote an interesting guest post for Search Engine People on the reciprocity approach to link building. This method is the traditional approach of pointing out 404s, links going to malware-infected sites and so on.

Melanie mentioned in passing that links to sites with drive-by-downloads of malware were some of the most valuable to identify.

People take these the most seriously and are especially grateful for having them pointed out. No one wants to send their visitors to sites where they’ll get viruses.

But how can you do that without visiting each site in turn? How can you speed up the process?

1. Introducing the Dr Web Anti-Virus Link Checker. It’s a Firefox extension that tells you if a page you’re about to load is dangerous or not. (Hattip Mashable’s security extensions list.)

2. People with Kaspersky subscriptions can click a red K icon in the upper right corner of their Firefox browser, and Kaspersky will scan all the links on the page. I don’t know what happens if they find one, or if they only alert you if they find a dangerous link, but so far I’ve yet to be alerted to anything…

3. On a similar note, you can do roughly the same with this anti-phishing extension that warns you of potentially harmful sites posing as legitimate ones. It tips you off before you fill in their forms, preventing you giving bad people your usernames and passwords.

Like this little bit of creative thinking? Get my book’s free chapter or add my rss feed to your reader.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

I recently read Rhys Wynne’s interpretation of John Mueller’s comment that it’s better to add content on your own site than to add it elsewhere. I disagree strongly that this is the right reading of John’s comment. There are numerous other readings of it:

- A bunch of links to a thin site (e.g. without valuable content) doesn’t make that site a positive user experience. So don’t guest post at the expense/neglect of your own site.
- Google is getting better at detecting quality content (and presumably differentiating it from junk).
- More in depth sites are favoured for now.

You can probably come up with your own. In any event, it’s clear that this is a whitehat tactic, and Google’s John Mueller was just pointing out that new sites need good content.

Bottom line: When a well-intentioned but perhaps sensational SEO bloggers says something, don’t take it for granted – check the source yourself. It turns out that the following [conveniently ignored] quotations were also part of the interview:

“Q 12. What’s your take on articles submission websites? I do a lot of article marketing & distribution for my clients. These are original articles written and distributed through sites like eZineArticles and iSnare. Obviously besides the exposure my clients get as experts, I am also looking at the SEO benefit of earning backlinks from these posts. How do you treat multiple copies of the same article spread over different sites?

“John Mueller: As mentioned in an earlier question, it generally makes much more sense to create great content for your own site, instead of giving it to a large number of other sites to publish.

“Personally, I would recommend not looking at it with regard to the links; think about how users will view the content and the people who created it. Having high-quality content on your own site will make it stand out much more than if that content is posted all over the web. If the content is unique and compelling, it will generally attract links naturally over time.”

Furthermore, there’s this goodie:

“Q 23. I know that inbound links will help my site’s ranking in Google search results but is that true for outbound links as well? I always link to quality websites from my articles where my visitors can read more about that topic but do these outbound links aid search rankings as well?

“Kaspar Szymanski: No, they don’t contribute directly towards your site’s rankings; however they add value for your readership and they contribute to the community, so feel free to continue this good practice. On the other hand, being selective and preferring quality sites to link to might help in how Google perceives your site.”

Linking out is a good practice for users, so long as the links go to quality sites. If their content is good enough for your site, doesn’t that make their site good enough for a link?

I note also that question 11, which Rhys cites to support his view, is in reference to new sites. This supports my reading about not sending visitors to thin sites:

“Q 11. I have launched a new blog and it obviously won’t rank in Google because none of the reputable blogs are currently linking to it. Therefore, I am actively writing guest posts on other blogs as that gives me a chance to get a link from them. Is Google fine with guest blogging and do links ‘earned’ from writing guest blogs matter?

“John Mueller: Making and promoting a new site takes time and effort. In general I would recommend putting that work into your own site, instead of creating content for other people’s sites.

“It’s much better to create great content for your blog and to let other sites refer visitors to your site on their own. Good luck!”

@Rhys: you’re a bright fellow who I’m sure is already having plenty of success and will see more in the future – no need to get a little extra attention at the risk of misleading a LOT of folks.

“Proving to be a huge traffic source actually. Sent me around 100 visitors already today.

Ironcially they don’t do pingbacks, but must’ve got about 50 or 60 scraped links from it O_o ” – [Rhys' comment on Problogger.com]

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

Question: Does it matter for my site’s SEO in Google or other search engines if I use www in the domain name? For example, do I need to use http://www.example.com or can i use http://example.com ?

Answer: It doesn’t matter at all to your SEO in Google or Bing whether you use www. in your domain name or not… with one big exception

Search engines don’t consider the use of www or not as a ranking factor, because it has no value in improving the relevance of their search results or improving the user experience.

There is one exception, however. You don’t want to use both a www version of the site and a non-www version.

If you do use both www and non-www, then you’ll have duplicate content problems, also known as “canonicalization” problems. The heart of the matter is that your content will be found in two separate locations.

(The word “canon” is used in religious terms to mean the “official set of holy writings.” That’s been adapted to other fields. So you can have the canonical set of Harry Potter books, which are the official books, and you can have the “canonical” version of a web page, which is the official one you want search engines to rank in their search results.)

What I mean is that if you have two identical versions of a page, one found at http://www.example.com and another found at http://example.com, and both show up fine to search engine spiders, you will have duplicate content.

This is problematic because a) It confuses search engines as to what is the canonical, or official, page. b) You will sometimes get links to one version and sometimes to another, such that you compete with yourself.

How do you do SEO successfully then, if you currently have such www vs non-www duplicate content problems?

You have two options.

1) Redirect either version to the other one using “301 permanent” redirects. There are numerous tutorials on this online, for DIY people, but you can also get a decent freelance programmer to do it for you. Personally, I get my hosting from Dreamhost and they give you an option to 301 redirect one version to the other.

I have a $50 off discount code at Dreamhost off any annual hosting package (or a $25 off discount on a monthly hosting package) if you need to set up a new site: Just enter GabSavesYou50 when you sign up for Dreamhost hosting.

(Full disclosure: It’s an affiliate link, but I’d endorse Dreamhost personally anyways, as I have in the past. Their tech support is super responsive and they have numerous tools to make things very easy, like one-click WordPress installs.)

2) Use the “canonical” tag to indicate to search engines which version is the official one. For those of you using WordPress, Yoast has a plugin to automate some canonicalization, but I’m not sure it’s applicable to this situation.

(3) Update: Malcolm Coles has a related article with examples of www vs non www that’s a valuable read on this topic.

If you liked this post, you might want to get my latest posts sent to you regularly, for free, by rss or by email.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

Legal fictions can be used by greyhat SEOs to overcome Google’s nearsighted paradigms on buying websites and buying links. While Google is trying to pass off its guidelines as law – and is succeeding in convincing some people of that fact* – the facts are otherwise, and my two-post series on the topic will show that the law can actually enable people to skirt Google’s techniques and expose its faults.

For my first post, I will address how legal fictions in the common law solve the problem of having a website’s SEO strength reset upon purchase and transfer. (I’ve yet to grasp how the civil law handles this, though my understanding is that it has similar mechanisms.)

Everyone’s favourite competitive webmaster, John Andrews, recently liveblogged Matt Cutts’ presentation at Domain Roundtable . Matt explained that Google eliminates all or nearly all of a site’s value when it is sold.

Obviously this is not the case for corporate America (Myspace anyone?), which in any case has its own legal fictions defending it (News Corp bought Myspace Inc, and Myspace.com was just one asset…). But for your average independent/competitive webmaster, the situation is different. So hopefully this will be helpful to you search marketing folks without the benefit of Fortune 500 status to back you up.

Update: The post is live: Buying Sites? Use Trust To Avoid Google Domain Demolitions.

For my second post, I will show how another legal fiction enables the purchase of text link ads under the radar.

After my post on hiding/disguising text link ads as AdSense and other ad format, and Shady’s point on making paid links look like the result of linkbait , this tip is another tool in the link purchaser’s arsenal.

Google is broken: it gave up the original PageRank paradigm of valuing links based on the traffic they send in favour of some public relations notion of links as “votes” (next SEO that repeats that links are votes is getting their ass kicked by yours truly). Until Google fixes itself, buying links from craptastic sites with no visitors (the majority of link buys, imho) is going to keep working.

*”[B]lackhat tactics are [...] illegal ways to game the system to your advantage.” – Avinash Kaushik’s Web Analytics an Hour a Day, p. 206; otherwise an interesting and educational read and one which I’ll be reviewing here once I’ve finished reading it. Note also: I spoke to Avinash before publishing this and he wanted to highlight two points of context for the above statement:

• First, he recognized that it was a mistake and poor choice of words. His point was merely that it’s unwise as your site might get banned (which is a more accurate statement and one I agree with).
• Second, he highlighted that he wrote that prior to working for Google, and thus it wasn’t a statement that they had him put out there. (That said, I’m quite certain their public relations department was more than happy to see that statement put out there.)

Liked this post on search engine law, greyhat SEO and Google? Subscribe to my feed.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

I’ve been asked the question recently in connection to business blogs: Should I blog on the company’s official site/domain name, or should I blog on on a fresh domain name? Each approach has its advantages, but with current search engine algorithms, my advice is to have the blog on the company’s main domain.

Blogging on the main corporate domain

Pros include:

• Traffic can transit over to the blog from the main company site and quickly build up the blog’s authority, share its posts and ideas, and connect to the company through its blogger(s). This assumes you don’t ignore your blog, like Facebook.
• Links go to the main domain that you’re trying to rank, which is key given the importance Google places on authoritative domains.
• If your blog links to inner FAQ pages or other things on the same site, the look and feel will be the same; the traffic is less likely to bounce upon arrival to the pages you linked to.
• Centralization if you have multiple company blogs (they can all go on one domain), as opposed to spreading them out over a disparate number of domains.

Cons include:

• You could get an extra source of “external links” by putting the new blog on a separate domain (and hosting on a different C block of IPs, at least for Google’s algorithms; hat tip Wiep’s link roundup).
• People may be reluctant to link to a “corporate” blog.
• If the bloggers become popular enough, they can just leave and blog independently.

Blogging on a new domain

Pros include:

• An extra source of links to your new site. I say source and note just ‘more links’ because my own experience with my old Montreal SEO site and this one is that after the first couple of links, additional ones aren’t very helpful, unless they’re deeplinks.
• It’ll be easier to attract links to the blog, though if the blog itself doesn’t convert visitors (either from the links or from the search engines), the value of this may be questionable.
• You might have a more relaxed graphic design, like being in the company lounge as opposed to the company press conference. This could, in turn, attract links and positive attention from CSS galleries and help enhance your company’s “cool factor” with web designers.

Cons include:

• More traffic will bounce when it hits the company’s official domain.
• Traffic won’t transit over from the main site as easily (plus you’ll have more bounces there too), so you won’t have as easy a time spreading the word at first.
• Links don’t go to the corporate domain, so you won’t build up its authority as easily.
• Doesn’t scale as well, inherently.

At the end of the day, the domain authority/strength issue is the big differentiator where I’m concerned. If you check out Wiep’s piece on how he had Matt Cutts’ site (about search marketing) rank for one of the most competitive phrases around, you’ll understand what I mean. Matt’s page had a single inbound link whereas players in that space usually have a pyramid of tens of thousands of links backing them up; the ranking was mostly due to his site itself having so many links and being such a trust authority.

Update: Romela of Business Solutions has a good addition in the comments. So does Dave, of Zoekmachine Optimalisatie SEO.

So keep those corporate / company / business blogs on the main company website. If you liked this, subcribe to my RSS feed.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

Google killed former SEOmoz CTO Matt Inman’s widgetbait because some Guardian reporter didn’t like it and wrote his negative opinion up. Then Aaron Wall was unlucky enough to trust a jerk who asked Matt Cutts about Aaron’s affiliate program based linkbuilding.

The question is: Will Amazon get a beat-down too? For their affiliate program based linkbuilding?

Don’t be silly.

3 Lessons for the Weary:

1) Don’t make Google look stupid. This is the cardinal rule of SEO. Spam isn’t a problem until it makes Google look bad. Sometimes that’s just by ranking. Other times you figure out a way to get lots of whitehat links and then bingo – you’re nailed. Note: I personally find the Widgetbait issue particularly weird, since Google’s saying to webmasters: we’re not holding you responsible for your own content.

2) Avoid getting too specific, particularly about your sites or clients’ sites.

3) Build your brand. Duh.

4) If you find some code that works to build whitehat, links in a scaleable fashion, keep it to yourself and do your best to stay on the downlow/practice reputation management. The Guardian folks wrote the story up a couple of times, and it’s plausible that they’d have wrapped it up after one piece with some reaching out.

Liked this? Subscribe to my RSS feed. (This link may be duplicated if you’re a new visitor by an automatic plugin. Just FYI if you’re wondering…)

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.

A question I’ve been asked a lot recently is “Can SEO be automated? Can I just get some software or hardware that’ll do the job automatically and get my site ranked at the top of the SERPs for my keywords?”

No, you can’t automate it.

First of all, search engine optimization’s foundation and core is about building links. Take the example of Google-bombing, where many people link to a certain webpage with the same anchor text, in a concerted attempt to have that page rank for the words in the anchor text, even though that page is not trying to rank for those keywords. This technique was succesfully used on several occasions and serves to demonstrate the power of link-building.

Now consider how you go about building those links. You contact webmasters and other people with websites and ask them to link to you. It helps a lot to be friendly and give them feedback on their own site, mentioning broken links or forms and other things that could be improved. And to a large extent, you want to go on to develop and maintain relationships with as many of those webmasters as you can, because that’ll get you other links in addition to that first one. Just consider what Marketing Sherpa’s Wisdom 2007 publication quoted me on (look at #27). You need to network to build links.

No matter how advanced spam-email software gets, it won’t network with other webmasters for you. It might let you include the url of the webmaster‘s site in your email, but you’re still not getting a link. First of all, because these things show you don’t care about the webmaster you want a favour from. So why should they care about you? The reaction goes something like: “Generic spam email? Delete. ”

So that’s why you’re never getting anything to automate link-building. Right there, you’ve just lost your chance at ranking for anything mildly competitive.

The second reason SEO can’t be automated is because the software that’s around is just plain garbage. I tried using one of these things just to do a site analysis of my own blog. I figured it couldn’t go wrong just reading code and telling me what was done well and what was done wrong. But this totally inept piece of software came back to me and told me it couldn’t find my keywords once on the page (I wish I remembered the name to help you steer clear but this was a looooong time ago – probably a year or more. Edit: I just saw more people asking this question on a forum I read. The software I tried was WebCEO.) . This was on my Montreal SEO blog, which, wouldn’t you know it, ranks for Montreal SEO!

Well, what about keyword research, you say? Keyword research is perhaps a minor exception to the rule here. You’ve got tools that will help you discover new keywords to try and rank for. But these are mostly for preliminary keyword research aimed at search engine marketing, or SEM. The distinction is important because if you just try and rank for the highest (or lowest) volume keyword in your niche, without having run an SEM campaign beforehand, you might end up ranking and waisting your bandwidth on unproductive traffic.

Imagine getting 10,000 visitors a day to your sports equipment website from people who are lazy and hate sports. Not only did you waste your time, effort and money achieving that ranking, but you’ve actually got hosting costs accumulating on top of it all!

Bottom line: SEO can’t be automated.

Downloads For RSS Subscribers:

• Content Marketing Cannon WordPress plugin - Consolidates content for more links, PageRank and traffic. Details: New SEO plugin: Content Marketing Cannon
• Internal Link Building, a WordPress plugin. It optimizes your internal navigation so you rank like Wikipedia. Details: Check the Internal Link Building update page.
• Get a free chapter from my upcoming book on advanced seo.
• Backlink Analysis With Majestic SEO's Data
  
• ROI on SEO Calculator spreadsheet.