Crawl.googlebot.com how and why ?

	Crawl.googlebot.com - how and why ?
	Discussion by yordan with 12 Replies. Last Update: May 3, 2009, 12:16 pm

yordan

I had a look at my private forum, and I saw a guest connected. I had a look at his IP address, did a whois, and found that this guest was
crawl-66-249-73-137.googlebot.com
How did this googlebot come to my private website ?
Is there a way to prevent these bots from crawling my website ?

Sun Apr 26, 2009 Reply New Discussion

hfbvm

just make a robots.txt file.robots.txt will work as you want. You can even disable crawling for specific pages, so you don't have to move your pages into a folder. Something like:

CODE

User-agent: *
Disallow: /index.php
Disallow: /main.php
Disallow: /inc/
Disallow: /img/
Disallow: /images/
Disallow: /js/
Disallow: /css/

And so on and so forth its easy more queries ask me maybe i could help you

Sun Apr 26, 2009 Reply New Discussion

yordan

Seems to be exactly what I want. Thanks, I'll try that way.

Sun Apr 26, 2009 Reply New Discussion

hfbvm

where's my rep for this?

Sun Apr 26, 2009 Reply New Discussion

yordan

QUOTE (hfbvm)

where's my rep for this?
Link: view Post: 136360

Your rep is here : http://www.astahost.com/index.php?s=&s...st&p=136357
Can't you really see it ?

Sun Apr 26, 2009 Reply New Discussion

zorba4

QUOTE (hfbvm)

where's my rep for this?
Link: view Post: 136360

I also see your rep, you said :

QUOTE

User-agent: *
Disallow: /index.php
Disallow: /main.php
Disallow: /inc/
Disallow: /img/
Disallow: /images/
Disallow: /js/
Disallow: /css/

does your browser have a problem? I use IE7

Sun Apr 26, 2009 Reply New Discussion

yordan

QUOTE (zorba4)

does your browser have a problem? I use IE7
Link: view Post: 136362

I use Firefox 2, and I see hfbvm's rep. Maybe he started sleeping ?

Sun Apr 26, 2009 Reply New Discussion

zorba4

QUOTE (yordan)

Maybe he started sleeping ?
Link: view Post: 136363

Or he forgot refreshing his browser window?

Sun Apr 26, 2009 Reply New Discussion

hfbvm

yes i can now see it sorry for the inconvenience or whatever.
I also use firefox.

Mon Apr 27, 2009 Reply New Discussion

aloKNsh

yeah i can also see....
and i use safari

Tue Apr 28, 2009 Reply New Discussion

yordan

QUOTE (aloKNsh)

yeah i can also see....
and i use safari
Link: view Post: 136433

Why safari ? Are you a Mac User ?

Tue Apr 28, 2009 Reply New Discussion

FirefoxRocks

QUOTE (yordan)

Why safari ? Are you a Mac User ?
Link: view Post: 136440

You do realize that there is also Safari for Windows. It is very similar to Google Chrome minus some features.

I'm using Firefox 3 on Ubuntu 9,04 at the moment. Just installed it today.

Sun May 3, 2009 Reply New Discussion

yordan

QUOTE (FirefoxRocks)

I'm using Firefox 3 on Ubuntu 9,04 at the moment. Just installed it today.
Link: view Post: 136596

Exactly what I said. Linux+windows people use Firefox.
Mac people send you mails with .websave files, and they a surprised learning that safari is not your default browser.

Sun May 3, 2009 Reply New Discussion