Value Links And Understand Search Like Matt Cutts With Submarine Crawling

Welcome Search Newz visitors! It seems that Search Newz’s syndicated version of my article, “If You Listened When Google Announced Submarine Crawling,” which follows up the one you’re seeing now, forgot to link to an important Matt Cutts video. So there’s the link to help you out. Anyways, on with the show – here’s what submarine crawling is all about, as interpreted from Matt Cutts’ explanations.

Matt Cutts’ post and this Webmaster Central post recently explained that “high quality” sites are being given special treatment – submarine crawling.

We all know that links from high-quality sites are more valuable than those from average or mediocre sites. Now, Matt and Google have given us a new measurement for finding high-quality sites – submarine crawling – and thus high-quality link prospects.

Russian Submarine







Russian Submarine courtesy of Orpheus Grey.

So WHAT is Submarine Crawling?

In essence, it’s self-motivated crawling with the purpose of discovering “deep” web content (hence me calling it “submarine crawling”). What I mean is that Google is proactively crawling your site in new and unique ways because it associates high-quality material with your own domain, and not because it’s reacting to a link pointing at your site, or any other signal.

Some parts of the web are poorly linked to, though they may have high-quality content. And since both humans and search engines discover new content based on links, these parts of the web can be invisible to the SEs.

(Aside: Hence another moniker for this part of the web: “The Invisible Web.”)

From a search engineer’s perspective, delivering relevant results featuring premium content is imperative. Therefore, if you know that a site has premium content, you can go back to it and try and mine it with submarine crawling to discover new premium content for your index.

It’s the 80/20 rule applied to search: 80% of your premium content comes from 20% of the web.

So knowing that Google is self-motivated (i.e. does not need external motivation like links) to crawl these sites. And specifically, to submarine crawl them for their deep content. Where SEOs are concerned, this is great news as far as attracting longtail traffic and evaluating the potential value of a link. (The text link brokers of the world had no – public – comment. 😉 )

  • Moral of the story 1: If you listen closely to what Matt and Google say, you can imply tips and ideas for your search marketing.
  • Moral of the story 2: Due to domain age factors and self-reinforcing rankings (where people google a topic, then link to what’s ranking), if you don’t achieve high-quality status with Google asap, optimizing for Google is only going to get harder as things move forward. The 80/20 rule is a self-reinforcing trend in search; that’s why Wikipedia is in the top 10 results for 970 of the top 1000 most popular keywords.

Further reading on Crawling, Link Value, and self-reinforcing rankings:

Wiep’s Link Value Factors is a good starting location. More recently, he’s also got this post on link building job interviews which touches on some of this stuff (his Internet Marketing company is hiring, FYI).

Bill Slawski has a new post (featuring more layman’s language, yay!!) on MSN’s treatment of old domains in ranking sites, in the context of a patent of theirs. An excerpt:

“A new patent application from Microsoft adds another twist, by also ranking domains based upon the ages of domains which link to those domains.

“This patent application assumes that newer domains have a ‘higher likelihood of being spam and/or being a part of a web farm that attempts to artificially inflate domain rankings for domains in the web farm.’”

(Meaning get started building your site’s trustworthiness in the SEs’ eyes before it’s too late. I note in passing that this post is about Google, but these points from MSN are valid regardless of engine, and the 80/20 rule in search is just reinforcing itself with every passing day.)

Jeff Quipp claims not to be an SEO but an “authority builder.” Sounds like someone’s ahead of the curve… got insider information you’d like to share Jeff 😉 ? That post was followed by “What is authority?” and Authority Building Tools.

My How to write robots.txt & Sebastian’s whole robots.txt categoryrobots meta tag category and his X Robots Tag category are also worth reading.

If you liked this, why don’t you add my RSS feed to your reader, or perhaps a link to this page?

Author: sroiadmin