Revealed: 6 Simple Reasons Why Search Engines Don’t Index All The Pages Of A Website

First let me tell you what one means by website indexing.

WebSite Indexing: Website indexing means announcing on the web about your own site. This can be done through uploading your website to various search engines. This refers to the number of pages that are uploaded on to the search engine for your website.

Formula for knowing how many pages of your website are indexed:

site: www.abc.com

This formula is useful for Google as well as for Yahoo. Why is this needed? This is needed to compare your website vis-à-vis competitors to find out which all pages are displayed of your website and whether they are decreasing in importance or are maintained at the same level in the search engines.

Why is there a reduction in the pages Indexed of a Website?

A website when it is new, & uploaded for the first time in the search engines, gets all the pages indexed by the crawlers the first time. This is based on the relevance factor. How relevant are the metatags with the content of the site. Now as the days pass by, that web crawler keeps on reducing your website pages from its database because;

1. Its database is limited.

2. It depends again on the relevancy factor. The webcrawler will eliminate all the irrelevant pages of your site vis-à-vis a new website (similar website) with a similar content. So, the webcrawler compares your set of keywords with the already established similar sites having a very good PR & a similar set of keywords. Now, if the crawler does not find certain keywords anywhere near the already established set of keywords, it will start reducing the number of pages.

Thus, the indexed pages get reduced. So, the measures that can be taken to prevent this are:

1. Check the Internal link structure of the website.

2. Check the keyword density.

3. If the uploading of the revamped site cannot be done at present, then resubmit the webpages with the new metatags into the various search engines.

4. Check the keyword competitiveness.

On page & Off page factors for a site also play an important role while indexing a website.

1.On page factors: Title, links, Description, Alt Tags, Bold Tags, Heading Tags, Keywords.If all the tags are used in a sufficient quantity, and not resulting to any spam, this also increases the website indexing.

a) Internal Link structure: This must be very strong. This shows, how good are your sites’ internal links. Whether they are all linked to each other & back. Whether there is any broken link or page under construction. If the Internal link structure of a website is good, it can retain a crawler or a customer to the same site. This also increases the indexing of the website.

b) Outbound Links: These are Links that is taking you out to another similar website. It is based on the page rank (PR) of a site. Word of caution: Outbound links are dangerous as they increase the popularity of the destination page(another site) but not your website. They decrease the indexing of your site. So, one should try his level best to retain a customer and not to let go the user to another website via an outbound link.

2.Off page factors:

a) Inbound links: Links coming in to your website. E.g. Reciprocal links or cross links. Here relevancy of the content plays the key factor. The content should be related to each other. Inbound links increase the popularity of your site. An Inbound link is never wasted. More, the inbound links, the better. One way links are also considered as an advantageous for your website. This also increases the website indexing.

b) Anchor text: Also called link text, referring to the text that appears in a text link. For example, if I link to abc website like this: Click here, then "Click here" is the anchor text for the link.

E.g. http://www.abc.com is the anchor text for ABC Corporation

Some search engines - notably Google - take anchor text into account when ranking websites. It is therefore beneficial to have keywords in the anchor text of links pointing to your site. This also increases the indexing of your website.

c) Directories: Getting links from directories like DMoz for getting it indexed. The search results displayed from the search directories are given a higher preference in search engines as compared to the normal search result.

d) Affiliates: Websites that are linking to you can take some share from you, as they are marketing a product for you. So, this also increases the website indexing.

e) Site Map: One must have a site map link in their website, so that all links can be pointed to it so that the website is indexed fast.

So, the factors mentioned above do play an important role in indexing of a website.