 Do Google Crawlers Index Our Sites? |
|
|
|
|
  |
 Aug 27 2005, 05:17 AM
Post
#1
|
|
|
Member [ Level 1 ]  Group: Members Posts: 43 Joined: 27-August 05 From: My House, Some Island, Earth, The Milkey Way Member No.: 8,129  |
Do the Google crawlers pick up info about all/most of the websites hosted here? If they do how frequently do they scan through the servers? Do all hosted sites include a robot.txt thingy or .htaccess (forgot what one google looks at)?
lol, my last site got on it but then my host started stuffing up my site so i stopped using it. QUOTE This post has been edited by microscopic^earthling: Today, 07:41 PM What was changed? This post has been edited by TheJeffsta: Aug 27 2005, 08:09 AM |
 |
 
|
|
Aug 27 2005, 04:28 PM
Post
#2
|
|
|
S.P.A.M.S.W.A.T. Group: Members Posts: 814 Joined: 22-January 05 From: San Antonio, Texas (No, I'm not dumb. I just moved here...) Member No.: 2,284 |
Crawlers crawl through links of sites they have already indexed, so Astahost links to your site, then Google should be able to crawl to your site. If you don't want them to, you can specify it in the "robot.txt thingy"
 , but you have to write it yourself, it is not included by default. I think they go through a site once a day or something, but I think you can set it to something else.
|
|
|
|
|
Similar Topics
|
Lo-Fi Version | Time is now: 8th October 2008 - 05:15 AM |