Site Indexing Under Google's Big Daddy

While Google has a strong focus on getting the best search results for users, it's getting tougher all the time to get better results for your website with Google. Many smaller businesses with limited resources are likely to find themselves at a disadvantage due to the constant algorithm changes.

At the end of March '06, Google completed another algorithm update called "Big Daddy". Algorithm changes are made on a regular basis by Google-sometimes twice a year-with the aim of getting the best results for Google search users.

What will this latest change mean to you and your website? For some of you it will make very little difference, for others it will impact considerably because the number of pages from your website listed within Google's search results will have diminished.

This article focuses on just one element of the March '06 "Big Daddy" update-the importance of getting external websites linked to yours, and how Google indexes websites across the internet.

Google uses link popularity as a measure to determine how often they will 'crawl' a website. Links have always been a strong measure of how Google ranks a website, but in the past this measure has been more geared towards determining PageRank and the order in which a web page is listed in Google search results.

What do I mean by indexing? Google uses an automated crawler called the Googlebot to find websites across the internet. For new websites that are not currently indexed by Google, Google relies heavily on finding these new websites through links from existing websites within its current search database. For existing websites the Googlebot visits websites already indexed within its database. If Google has already previously indexed a website, then it will generally be faster to index other pages within your website.

Google has introduced an additional complexity through its Big Daddy update. One thing many site owners have noticed since the end of March is that many pages from their websites have vanished from Google's index. Google have said that with this recent update they are placing more weight on 'crawling' external links from websites that they classify as being important. The more external links you have pointing towards various pages within your site-not just your home page-, the more likely Google will crawl these pages and find other pages within your website.

Google's webmaster guidelines state:
"In general, webmasters can improve the rank of their sites by increasing the number of high-quality sites that link to their pages." http://www.google.com/support/webmasters/bin/answer.py?answer=34432&hl=en

Why are quality links so important?
If Google doesn't 'crawl' all your pages, then you will have no chance of having all your website content appear in Google's search results. This means that you may have important information within your website that Google users will not find. At one stage it was enough to have your home page indexed and the rest of your website through a user-friendly website structure that the 'crawler' could follow. In the new environment however, that is no longer enough.

The best way to find which pages of your website Google has within its database is to type "site:www.yourdomain.com.au" into Google. This will bring up a listing of all your website results in Google's index.

Note:." To show all of your results you will most likely need to click on the link of the last page that says "repeat the search with the omitted results included"

External links pointing towards every single page within your website are not necessary, however make sure that external links point towards important pages within your website, and that your website structure is well linked to other parts of your website. The Googlebot can easily crawl other pages from there.

An easy to follow navigation structure is vital for search engines, but more importantly for human visitors. Make sure that each page within your website has a clear link to the most relevant pages within the navigation structure. These individual pages should have clear links to relevant sub-pages. This way, you are satisfying both your visitors who are the most important when it comes to your website and the search engines.

Basic and important elements to help your site get indexed:

Create a sitemap with all your website pages listed.
Keep the appearance of this page simple. The simpler it is, the less code a search engine needs to filter. Example sitemap: http://www.sydney-directory.com.au/sitemap.php

Create a Google Sitemap.
Google sitemaps make it easier for your site to be picked up by Google, although it does not guarantee inclusion within Google's index. https://www.google.com/webmasters/sitemaps/

A Clear website navigation structure.
The larger a website becomes, the more difficult it is to manage. A clear website navigation structure is achievable, although it will take careful planning and organising. Microsoft is a perfect example of a large website with a simple, easy to follow navigation structure. www.microsoft.com

For a business with limited resources the constant algorithm changes can be quite frustrating, however if you focus on building relevant links to your website as well as relevant content, when the algorithm changes are made, then any changes will have less impact on your website.

This article doesn't cover everything that has changed in the recent Big Daddy update; however you need to be aware of this important change. If you have the time and patience, you can find out more information on these recent updates in Matt Cutts' blog. Be prepared for some serious reading time as this is a very long post. http://www.mattcutts.com/blog/indexing-timeline/

If your website layout is not effective, contact your web designer or SEO specialist to find out how they can help improve your website layout for both your website visitors, as well as the search engines.