Loading...


bookmark - Meta Tags To Stop Spiders

Meta Tags - To Stop Spiders

 
 Discussion by jcguy with 17 Replies.
 Last Update: January 13, 2008, 8:44 pm
 
bookmark - Meta Tags To Stop Spiders  
Quickly Post to Meta Tags To Stop Spiders w/o signup Share Info about Meta Tags To Stop Spiders using Facebook, Twitter etc. email your friend about Meta Tags To Stop Spiders Print
Reply / Comment New Discussion / Topic Share / Bookmark E-Mail a Friend Print

Sometimes you don't want search engine spiders to spider your site. For example, you may not want your pages to show up in a google search. Here is what you can do: use meta tags in your web pages. You can use meta tags to control indexing and crawling of your site.

By default, every single page in your site will be indexed by search engine spiders. To control this default action, just use meta tags!

How? Your meta tags must be located in the HTML codes of your pages, in the header between the <head> and </head> tags.

So if you want spiders to index your page and follow every link on it, insert:
<meta name="robots" content="index, follow">

For no indexing but following of links, use:
<meta name="robots" content="noindex, follow">

So more combinations are possible, example:
<meta name="robots" content="index, nofollow">

and:
<meta name="robots" content="noindex, nofollow">

Let's say you don't want your pages to be indexed but want spiders to follow links on it, your HTML should starts like this:

<html>
<head>
<title>Page title</title>
<meta name="robots" content="noindex, follow">
</head>
<body>body contents</body>
</html>

That's it! Spiders will always check for such meta tags before deciding waht to do with your pages.

Hope this helps :)

   Mon Sep 6, 2004    Reply         

that is very interesting to know jcguy, there are tons of things that you can do with meta tags which i have heard about before, they can be very useful for people in certain situations.

This topic seems very familiar to me, were you at Inuration Technologys at all? cos this and that UNI.CC tutorial are familiar to me?

   Tue Sep 7, 2004    Reply         

If you're using XHTML, be sure to close the meta tags with a / at the end of each tag, like this example:

CODE

<meta name="robots" content="noindex, nofollow" />

If having web compliant HTML or XHTML is important to you, also make sure you declare a DOCTYPE in the <html> tag.

   Fri Sep 10, 2004    Reply         


Does anyone know if there is a way to stop leeches from attaching themselves to files and stealing the bandwidth? could it be done with a specific meta tag or is something else required?

   Fri Sep 10, 2004    Reply         

another way to block spiders from visiting and indexing your website if you have your own domain is to create a file called "robots.txt" in the root directory of your website. put this in the text file -

user-agent: * Disallow: /*

- this means that all spiders that are reading this file should not visit or index anything at all.

with this file, you can also actually specify which directories and files you do not want the spiders to index, e.g.

user-agent: * Disallow: /index.html
Disallow: /animals/*
Disallow: /objects/toaster.html







   Sun Sep 12, 2004    Reply         

Ohhh wow that's really interesting! =) overture, there is cPanel provided by asta host... I was at inuration so i;ve used cpanel. I think its called Web Protect (htaccess editor)? I'm not too sure...

   Sun Sep 12, 2004    Reply         


QUOTE (overture)

Does anyone know if there is a way to stop leeches from attaching themselves to files and stealing the bandwidth? could it be done with a specific meta tag or is something else required?
[post="1090"]<{POST_SNAPBACK}>[/post]


i assume you're talking about images. here's what you can do -

1. create a separate folder and put all your images (or the images you wish to protect) in it.

2. create a text file called ".htaccess" in the above folder.

3. the text file should contain these lines -

SetEnvIfNoCase Referer "^http://www.blah.com/" locally_linked=1
SetEnvIfNoCase Referer "^http://www.blah.com$" locally_linked=1
SetEnvIfNoCase Referer "^http://blah.com/" locally_linked=1
SetEnvIfNoCase Referer "^http://blah.com$" locally_linked=1
SetEnvIfNoCase Referer "^$" locally_linked=1
<FilesMatch "\.(gif|jpe?g)$">
Order Allow,Deny
Allow from env=locally_linked
</FilesMatch>

- and replace blah.com with your domain name.

4. now when someone tries to steal bandwidth by linking to your image from their site, the image will not be displayed.


if there are other files that you want to protect e.g. zip files, just put them in that special folder and add the zip extension to the FilesMatch line -

<FilesMatch "\.(gif|jpe?g|zip)$">

- that's it!

hope this helps :)

   Mon Sep 13, 2004    Reply         

The best way to stop search engine spiders from snooping on your website is to use a robots.txt file, but if you want to protect a webpage that you have limited access to (i.e., a simple blog like LiveJournal), then you can use the meta tag alternative.

   Mon Sep 13, 2004    Reply         

dissipate, i was talking about leeches witch attach themselves to downloadable files which when clicked use up your bandwidth, as i know someone who had 20 gigs of bandwidth used in a few days due to them. i was wondering how to stop them, would the way you suggest work for downloadable files, like .zip/.rar/.exe...

edit:

would i have to change the extension types, like you have put gif|jpe?g

   Mon Sep 13, 2004    Reply         

most download sites use server side scripts to verify the referer if the referer is not from them it won't download anything :)

   Mon Sep 13, 2004    Reply         

overture: hmm i'm not really sure what you mean sorry. could you elaborate further? do you mean you're putting up downloadable files like zips, rars, exes and you don't want people to download them or something?? *boggle*

   Mon Sep 13, 2004    Reply         

So is there a tag that makes it more likely to index your site? For example, if I google my name, I'll find my lame college personal websites well above my current websites. I don't want people to see my embarrassing stuff from back in the day! :)

   Thu Sep 16, 2004    Reply         

hehe i think you have to work on your page ranking then :)

   Thu Sep 16, 2004    Reply         

What are spiders and leaches? Are spiders or leaches bad? If so what do they do?????

   Fri Sep 24, 2004    Reply         

interesting topic, jcguy, i think it's great to know about the usefullness of meta tags, since there are so many!

   Sat Sep 25, 2004    Reply         

QUOTE (KyoNiwa)

What are spiders and leaches?  Are spiders or leaches bad? If so what do they do?????
[post="3634"]<{POST_SNAPBACK}>[/post]


Spiders are actually robots that crawl the web constantly by following links on a webpage to other pages. Because the web is made up of billions of web pages linked together, it is possible for the spiders to crawl the entire content of the web. That's how Google, with its spiders, can build a database for you to search.

As for leaches, I'm not very sure, but it sounds very scary

   Mon Sep 27, 2004    Reply         

Well, in this case, its spelt 'leeches'...
Leeches are also automated robots or programs that search websites for files and mass-download them, eating up bandwidth. Leeches are also programs used by people on P2P networks to download files from a specific network without letting others access file from the leech....

   Mon Sep 27, 2004    Reply         

get around this
Meta Tags

If you want to use google to find pages like this, type this in the search box

"robots.Txt" disallow:" filetype:txt

-ladyzfirst

   Sun Jan 13, 2008    Reply         

Quickly Post to Meta Tags To Stop Spiders w/o signup Share Info about Meta Tags To Stop Spiders using Facebook, Twitter etc. email your friend about Meta Tags To Stop Spiders Print
Reply / Comment New Discussion / Topic Share / Bookmark E-Mail a Friend Print

Similar Topics:

What Are Meta Tags And What Do They...

What are all the meta tags and what exactally do they do? ...more

   24-Sep-2004    Reply         

Html Meta Tags Tutorial

So, you've got your beginning page started, your homepage is defined, neat, and it looks like everyone will love it - STOP. Do you really think your site will be even accidently view by anyone, even if you do have a bought domain?! Don't kid yourself; the fact of the matter is, you' ...more

   09-Sep-2005    Reply         

All On Meta Tag The Mistry

To create the meta tags for your site, you will be concentrating on your site description and site keywords. To create your site meta tag keywords, try copying and pasting your homepage content into MS Word. Delete all insignificant words (like “a” and “the”). Then delete all punctuati ...more

   17-Dec-2007    Reply         

(13) Converting HTML over to XHTML Crossing over to the darkside  Converting HTML over to XHTML Crossing over to the darkside