Google’s Big Daddy; Latest updates

In other musings recently on Google I had questioned the stability of Gs infrastructure and it’s storage capabilities being part of the whole ‘Big Daddy' storm. In response to my original conspiracy theory, Google’s Matt Cutts has recently said, “First, I believe the crawl/index team certainly has enough machines to do its job, and we definitely aren’t dropping documents because we’re “out of space.””

On issues related to many pages being dropped or gone supplemental, he penned, “The second point is that we continue to listen to webmaster feedback to improve our search. We’ve addressed the issues that we’ve seen, but we continue to read through the feedback to look for other ways that we could improve.”

So, as always, who really knows. Even the folks at G seem to be struggling with unexpected side effects of the new algorithm. I must say our web stats work (analytics) lately has shown a marked drop in many factors as related to marketing to Google. They are completely different than the other majors, (Yahoo,MSN,AOL) for relative results in areas such as external links, page rank, indexed pages and more.

Factors affecting ‘dropped’ pages in the Google index.

Linking strategies; Some great efforts have been made by Matt to explain things better. Linking weights have been given an overhaul with Big Daddy. Some dropped pages will undoubtedly be lost do to the quality of the inbound/out bound links, (can you say link farms?). On sites that had dropped pages Matt stated that they, “were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling.”

Supplemental results; Another issue was the fact that Big Daddy is, for the most part, a new technology. It does not draw from the existing Google archives (called supplemental results). It is indexing the web on it’s own. That mean many pages on sites such as forums that have been indexed for years, are not any longer. “The Bigdaddy update is independent of our supplemental results, so when Bigdaddy didn’t select pages from a site, that would expose more supplemental results for a site.- I worked with the crawl/index team to tune thresholds so that we would crawl more pages from those sorts of sites.”, said Guru Cutts. They have been working since the end of March to improve in this area.

Upon Review

Matt also did some reviews of various sites that had complaints with how the index was weighting them. One was a real estate site, about a Eastern European country. It illustrates the risks of bad link partners. The site was linking to a free ring tones site, an SEO contest, and an Omega 3 fish oil site. As Cutts put it, “I’d think about the quality of your links if you’d prefer to have more pages crawled. As these indexing changes have rolled out, we’ve improving how we handle reciprocal link exchanges and link buying/selling“. They are looking for more ‘organic’ linking patterns. It is n attempt to cut down on SE manipulation.
Then came another real estate site. The owner says that they used to have 10K pages indexed and now they have 80. Here he found links to mortgages sites, credit card sites, and exercise equipment. Same problem with this one. Questionable linking partners means questionable resource.

Lessons to be learned

The whole ‘supplemental’ results part doesn’t bother me greatly. My site, as well as many I manage SEM for, had pages listed in the SR (supplemental results) that were no longer even on the site. There may be fewer ‘total pages’ indexed, but I really only want the relevant ones in there anyways. Even the linking strategies changes don’t concern me. I have never been a fan of ‘wild eyed’ linking strategies to sites that would refer non-targeted traffic. I want conversions. If it’s sales or sign ups, my goal is to garner traffic that is meaningful.

More on big Daddy; Big daddy; Big Daddy; The ever changing Algorithm landscape , Big Daddy and supplemental results ,