How to Index Your Website

Posted on 25th Apr 2013 @ 9:11 AM

                                                                                   How to Index Your Website

How to find out which pages of your site are indexed?
One way to understand this is to type into your browser site: domain. This method works with Google, Yahoo and Bing. Search engine will display a list of pages on your website that are indexed. If you find that page on your site is not indexed, and you're not restrict access to the robot to its purpose by robots.txt or meta tag, the search engine probably does not know that this page does not exist or access to it.

What should I do to make search engines index your entire site?
If you find that the pages of your site are indexed by search engines, do the following:

1. Check that the search engines can access this page
Check your robots.txt file and ensure that it complies with the rules. Many webmasters inadvertently block parts of your site by search engines due to an error in the text of robots.txt.
You should also make sure that the meta tags of the page you do not block the crawlers from accessing it. This can happen if you have a meta "noindex" tag on the page and you forgot to remove it.

2. Create a sitemap
Major search engines - Google, Yahoo, Bing, maintain a site map. This is not a site map you've seen in various websites. These site maps are created to facilitate the orientation of visitors to your site to its resources. Sitemap protocol is designed for search engines, not people.
The site map must have a certain format. More details can be found on sitemaps.org.

3. Make changes to the robots.txt indexing site map
Once you create a sitemap and upload it to the server, plug in robots.txt following order:
Sitemap: http://www.example.com/name.xml
where www.example.com is the name of your site, the address must correspond to the actual location on the map on your site.
Search engines that visit your site automatically check your robots.txt file before you crawl your site. When you read the file, you will see the site map and load it for more information. This will find all pages that have missed the first time and give them indexed.