jcguy
Sep 6 2004, 07:33 AM
Sometimes you don't want search engine spiders to spider your site. For example, you may not want your pages to show up in a google search. Here is what you can do: use meta tags in your web pages. You can use meta tags to control indexing and crawling of your site. By default, every single page in your site will be indexed by search engine spiders. To control this default action, just use meta tags! How? Your meta tags must be located in the HTML codes of your pages, in the header between the <head> and </head> tags. So if you want spiders to index your page and follow every link on it, insert: <meta name="robots" content="index, follow"> For no indexing but following of links, use: <meta name="robots" content="noindex, follow"> So more combinations are possible, example: <meta name="robots" content="index, nofollow"> and: <meta name="robots" content="noindex, nofollow"> Let's say you don't want your pages to be indexed but want spiders to follow links on it, your HTML should starts like this: <html> <head> <title>Page title</title> <meta name="robots" content="noindex, follow"> </head> <body>body contents</body> </html> That's it! Spiders will always check for such meta tags before deciding waht to do with your pages. Hope this helps 
Reply
overture
Sep 7 2004, 02:48 PM
that is very interesting to know jcguy, there are tons of things that you can do with meta tags which i have heard about before, they can be very useful for people in certain situations. This topic seems very familiar to me, were you at Inuration Technologys at all? cos this and that UNI.CC tutorial are familiar to me?
Reply
zarjay
Sep 10 2004, 01:02 PM
If you're using XHTML, be sure to close the meta tags with a / at the end of each tag, like this example: CODE <meta name="robots" content="noindex, nofollow" /> If having web compliant HTML or XHTML is important to you, also make sure you declare a DOCTYPE in the <html> tag.
Reply
overture
Sep 10 2004, 01:53 PM
Does anyone know if there is a way to stop leeches from attaching themselves to files and stealing the bandwidth? could it be done with a specific meta tag or is something else required?
Reply
dissipate
Sep 12 2004, 02:38 PM
another way to block spiders from visiting and indexing your website if you have your own domain is to create a file called "robots.txt" in the root directory of your website. put this in the text file - user-agent: * Disallow: /* - this means that all spiders that are reading this file should not visit or index anything at all. with this file, you can also actually specify which directories and files you do not want the spiders to index, e.g. user-agent: * Disallow: /index.html Disallow: /animals/* Disallow: /objects/toaster.html
Reply
currahee
Sep 12 2004, 03:48 PM
Ohhh wow that's really interesting! =) overture, there is cPanel provided by asta host... I was at inuration so i;ve used cpanel. I think its called Web Protect (htaccess editor)? I'm not too sure...
Reply
dissipate
Sep 13 2004, 09:53 AM
QUOTE(overture @ Sep 10 2004, 09:53 PM) Does anyone know if there is a way to stop leeches from attaching themselves to files and stealing the bandwidth? could it be done with a specific meta tag or is something else required? i assume you're talking about images. here's what you can do - 1. create a separate folder and put all your images (or the images you wish to protect) in it. 2. create a text file called ".htaccess" in the above folder. 3. the text file should contain these lines - SetEnvIfNoCase Referer "^http://www.blah.com/" locally_linked=1 SetEnvIfNoCase Referer "^http://www.blah.com$" locally_linked=1 SetEnvIfNoCase Referer "^http://blah.com/" locally_linked=1 SetEnvIfNoCase Referer "^http://blah.com$" locally_linked=1 SetEnvIfNoCase Referer "^$" locally_linked=1 <FilesMatch "\.(gif|jpe?g)$"> Order Allow,Deny Allow from env=locally_linked </FilesMatch> - and replace blah.com with your domain name. 4. now when someone tries to steal bandwidth by linking to your image from their site, the image will not be displayed. if there are other files that you want to protect e.g. zip files, just put them in that special folder and add the zip extension to the FilesMatch line - <FilesMatch "\.(gif|jpe?g|zip)$"> - that's it! hope this helps :)
Reply
zarjay
Sep 13 2004, 02:31 PM
The best way to stop search engine spiders from snooping on your website is to use a robots.txt file, but if you want to protect a webpage that you have limited access to (i.e., a simple blog like LiveJournal), then you can use the meta tag alternative.
Reply
overture
Sep 13 2004, 04:20 PM
dissipate, i was talking about leeches witch attach themselves to downloadable files which when clicked use up your bandwidth, as i know someone who had 20 gigs of bandwidth used in a few days due to them. i was wondering how to stop them, would the way you suggest work for downloadable files, like .zip/.rar/.exe... edit: would i have to change the extension types, like you have put gif|jpe?g
Reply
r3d
Sep 13 2004, 05:15 PM
most download sites use server side scripts to verify the referer if the referer is not from them it won't download anything
Reply
iGuest
Jan 14 2008, 04:44 AM
get around this
Meta Tags
If you want to use google to find pages like this, type this in the search box "robots.Txt" disallow:" filetype:txt -ladyzfirst
Reply
Eric Straven
Sep 27 2004, 03:56 PM
Well, in this case, its spelt 'leeches'... Leeches are also automated robots or programs that search websites for files and mass-download them, eating up bandwidth. Leeches are also programs used by people on P2P networks to download files from a specific network without letting others access file from the leech....
Reply
jcguy
Sep 27 2004, 11:38 AM
QUOTE(KyoNiwa @ Sep 25 2004, 04:02 AM) What are spiders and leaches? Are spiders or leaches bad? If so what do they do????? Spiders are actually robots that crawl the web constantly by following links on a webpage to other pages. Because the web is made up of billions of web pages linked together, it is possible for the spiders to crawl the entire content of the web. That's how Google, with its spiders, can build a database for you to search. As for leaches, I'm not very sure, but it sounds very scary
Reply
kraizii88z
Sep 25 2004, 05:55 PM
interesting topic, jcguy, i think it's great to know about the usefullness of meta tags, since there are so many!
Reply
KyoNiwa
Sep 25 2004, 04:02 AM
What are spiders and leaches? Are spiders or leaches bad? If so what do they do?????
Reply
Recent Queries:--
site:www.astahost.com meta-tags_t186 - 10.79 hr back. (1)
-
ddboard hack - 48.90 hr back. (1)
-
stop spyders indexing page - 98.59 hr back. (2)
-
meta tag stop spider - 110.33 hr back. (1)
-
insert google meta tag into livejournal blog - 123.64 hr back. (1)
-
should i add a zip code meta tag - 254.20 hr back. (1)
-
meta tag to stop spiders - 268.67 hr back. (1)
-
meta tag referer - 281.25 hr back. (1)
-
meta to stop indexing - 307.57 hr back. (1)
-
"disallow" filetype: zip - 308.79 hr back. (1)
-
head tag for referer - 329.32 hr back. (1)
-
add php meta tags - 338.27 hr back. (1)
-
http referrer meta tag - 353.22 hr back. (1)
-
waht are meta tags - 385.82 hr back. (1)
Similar Topics
Keywords : meta, tags, stop, spiders
- Stop 0x00000024
(5)
How To Stop A Dog Barking?
(7) Hello, We have a dog and I cant stand it barking all night.. I am afraid to disturb my neighbors..
Do you know any little tricks to help me to stop it from barking ? ** I fed him I do everything..
But it is still barking.. Thanx....
Ideas Wanted: Car Oil Bottle
How do I stop it falling over and leaking? (2) Okay Einsteins... I have a small problem I'm getting sick and tired of. Whenever I buy a Castrol
4L plastic bottle of car oil, the lid is really hard to take off (you have to break the ring, and y
fingers have nerve damage making this quite a feat because I can't grip well) and has foil
underneath... but once this is broken, the lid comes off all too easily. If I have the car in my
boot, the bottle often tips over and I find the oil has leaked everywhere. It's not cheap and I
ruin anything it comes into contact with. Short of paying someone a stupid amount o....
Stop Hold Of Hosting Credits
vacation (6) I will be out of net access for a while, and I want to put a stop hold on the credits as I don't
want to lose them because I can't post. I'm also thinking of taking the site down and
making a new one so I guess I have to replace them. So the big question is how do I do that? How
do I stop my credit usage? Don't care if the sites down as I want to make a new one.... Also
is there a way to replace the old site domain name with a new one and start all over??? Another
question what happened to all my hosting credits? I have a lot and when I became a ....
Style P And H? Html Tags
(2) Well, I have found that css was very hard to predict and precise control their apperance. When mix
and use the p and h? tags. Their apperaence were not identity. Place the h? tags within the p tag
causes style error on both firefox and IE. So, anyone should do it reserved, instead, place the p
tag with any h? tags will do the jobs on both browsers. Still learning it coz it much varies when
doing styles. Tested it on XHTML 1.0 Strict....
Opensuse: Stop Asking Root Password
(5) I've just installed openSuse 11.0 on my laptop to replace Vista. However, it's very annoying
that it constantly keeps asking for a root password when I want to configure something. I even
changed my default group from 'users' to 'root', but that didn't help , it keeps
asking for a password /mellow.gif" style="vertical-align:middle" emoid=":mellow:" border="0"
alt="mellow.gif" /> . I've also tried adding /sbin/yast2 and other to the sbin configuration,
but that doesn't work either (because for example, yast2 isn't launched with sud....
All On Meta Tag The Mistry
Meta Tag (0) To create the meta tags for your site, you will be concentrating on your site description and site
keywords. To create your site meta tag keywords, try copying and pasting your homepage content
into MS Word. Delete all insignificant words (like “a” and “the”). Then delete all punctuation and
words/phrases that have been repeated. This is your initial meta tag list.Now fine-tune the list.
Add more terms and delete less relevant ones. Do not use commas or other punctuation. Add your name
and URL (www.yourdomain.com) because sometimes people type those into the search eng....
10 Extremely Useful Websites To Stop Big Brother From Snooping On You
(5) 10 Extremely Useful Websites to Stop Big Brother From Snooping on You I just read this excellent
article that i think could be helpful for all of us, as it's name says it shows up 10 of the
most useful website to stop spam, junk mail, etc. From the article: QUOTE If you are tired of
receiving junk mail, spam and annoying telemarketing phone calls, then this list is for you. If you
desire to take steps to stop the snooping from the government, hackers and marketing agencies, then
this list will show you the way to privacy freedom. People probably don’t reali....
The Spider Catcher
Catching spiders in order to free them in your garden (5) Have a look here, it's really funny : http://www.nierle2.com/s01.php?shopid=s01&...aa&bnr=4953
Besides the text describing the device, you will see a little movie showing how it works. It's
really great ! Unfortunately it costs 19 euros, else I would already have bought it ! Until now,
each time I see a spider, I take my shoe in order to destroy it : this makes a big splat on the
ceiling, and it's very bad because it destroys a life. With this device, you take the spider
away from your keyboard (where it's not really useful) and put it two kilometers a....
A Gaiaonline Problem
A problem i want to stop (9) I have a problem getting on Gaiaonline. When I go it always says "SLOW DOWN Slow down, hotshot!
You've been loading pages too fast, and our system doesn't like it. Take a break for a few
minutes so we can catch up! This is not a ban. Come back in a littlw while, ok?" and I want it to
stop! So can anyone please help me out here?....
Google Spiders Scan Astahost Daily
lovely advertising (3) Hello , I had some troubles with my site at the begining of hosting so I posted these couple of
questions here and got the answers , almost solved all my problems /smile.gif"
style="vertical-align:middle" emoid=":)" border="0" alt="smile.gif" /> now my site is working fine
so I used google submit URL for it and it didnt /wink.gif" style="vertical-align:middle" emoid=";)"
border="0" alt="wink.gif" /> yet , now everytime I google my site the search results are from
astahost all like that 1-my site doesnt work help me 2-www doesnt work what can i do
3-advanced fr....
Game: Stop Stop -> How Timely Are You?
(6) Out of boredom, I turned one of the games that we used to play some times into a .NET application.
The game named Stop Stop is pretty simple in concept where you need to stop the clock as close as
possible to the checkpoints 10, 20, 30, 40 & 50. You earn scores based on how close you reach to the
checkpoints. It maintains an online record of the scores obtained by players which can be seen by
anyone along with the offline scores. Here's the link to it:
http://maxotek.net/products/list.php?name=MX%20Stop%20Stop Here are some screenshots:- Main
Window Inter....
Stop Smoking, Save Thousands!
Literally, it is possible. (51) Last semester I was lucky enough to come across one of those teachers that knows what you want to
learn and has the perfect approach to teaching it to you. Have you ever had that expirience? It is
simply awesome. Anyways, lets get down to buisness. As an assignment in this class I took, we were
supposed to calculate how much money a person who smokes 1 pack of cigaretts a day would save if
they quit smoking and invested their money into a fund with a 9% intrest rate. First thing we had to
do was calculate how much a pack of smokes would cost on average, and where I liv....
Can Search Engine Spiders See Dynamic Content?
(1) I want to make a web page using PHP codes to require() a list of keywords to put in between the
meta keywords tag. If I did this, would the search engines be able to see these PHP generated
keywords, thus making it more SE friendly?....
HTML Tags
Ever wondered what a HTML tag was (4) Many people that i have met have mentioned HTML to me without really knowing what it really means.
Because of this i thought that maybe it would be a good idea to compile a tutorial for everyone to
read. When i listen to people talking about HTML; they never seemed to know what a HTML tag was so
that is what i will focus on here. ------------------- Some people who use HTML may notice that I
place the tags below in the order they appear in a document. Most people who use HTML know the tags
which no HTML document can do without. Those are: = this is the tag which ev....
Sparkx? Are You There?
Sparkx is here! So stop reading and start greating me! lol (0) Hey eveyone. How is it going? /cool.gif" style="vertical-align:middle" emoid="B)" border="0"
alt="cool.gif" />....
Price Tags
(5) can somebody here tell me the price of assembling parts of computer iam from india but iam not aware
about computer prices and i want to buy a laptop so please tell me what sould i required for a
complete laptop i mean notebook thanks waiting for suaitable reply friends........
Do You Know Of Any Hide Tags For vBulletin ?
(1) Currently there are no known hide tags available for vBulletin Version 3.6.0 Goldand our site
desperatly needs some to stop media being stolen from our site. I would like someone to try and
develop soem hide tags that require you to reply to the post to see the hidden links. Could someone
please try this or if they know about any hide tags that exist for vBulletin Version 3.6.0 Gold
please post the link here it would be very helpful. Maybe trying to alter previous hide hacks may
work?? Yours truly, zacaroo....
A New Meta Tag
(1) Google crawlers sometimes uses ODP to get the description of your webpages. Unfortunately ODPs do
not update/refresh contents as frequently as most of us would like it to have. Now Google has
introduced a new META Tag to direct search engine crawlers not to use information available in the
ODP. This is the META Tag: CODE To specifically direct GoogleBOT, you need to use :
CODE See : Inside Google Sitemaps Does not it indicate that Google is showing less
trust for ODPs ? Regards, Sid....
Profusion.com: The Original Meta-Search Engine
The original Meta-Search engine (9) I know all of you are busy using (and enjoying) Google's SE as your favourite one, but you
would "think again" /huh.gif" style="vertical-align:middle" emoid=":huh:" border="0" alt="huh.gif"
/> after reading this post... The Meta-Search Technology Profusion.com is one of the free
meta-search engines available on the internet. Basically, here's what it does - "Profusion.com
searches multiple Search Engines (like Google, Yahoo, MSN etc.) simultaneously for the search
query you enter. Then it compares all results from the various Search Engines and t....
IE And It's Annoying NBSP (Non-Breakable Space)
How to stop it (2) Fellow designers, developers and table makers. This is a short explanation and tutorial on
problem-child Internet Explorer's urge to fill empty div or table tags with a non-breakable
space (html char code ). Why does IE put NBSP into empty div/table tags? Here comes a
really professional answer. I don't know why. Surely it takes that bit longer to put a
conditional statement that if there is an empty tag, fill it with somet. Surely, it would be easier
to say don't fill. But then again, MS never do things by half. The Work Around There may be a
Ja....
Ever Needs To Find Out A Table Height Or With With JavaScript
Well stop looking, here is the answer (CSS and JS) (2) Welcome everybody to this litle tutorial. by v.DragonEyE.n09 Introduction: Using
javascript you can find the height and width of a table, cell, div, image, etc.. the more simple way
is to ask for this... QUOTE id= "myElement" border= "0" cellpadding=
"0" cellspacing= "0" style= " height : 300px ; width
: 450px ; " > some
fake text and images for the example some fak....
Ruby - Better Programming Language Than Java Or C#?
Ruby is an object-oriented, meta-programming language that has many de (1) hey all i read this very intresting and intriguing article and i thought i should share it here..
find example codes in the attached file. cheerz! shiv Ruby —A Diamond of a Programming
Language? QUOTE Have you heard about Ruby? It seems to be a popular topic in software
development circles today. This language came to my attention at a Java conference last spring,
where gentlemen like Bruce Tate (author of Bitter Java and Better, Faster, Lighter Java), Dave
Thomas (Pragmatic Programmer: From Journeyman to Master), and others were all talking about Ruby an....
Html Meta Tags Tutorial
-What it is/does; how to utilize it- (21) So, you've got your beginning page started, your homepage is defined, neat, and it looks like
everyone will love it - STOP. Do you really think your site will be even accidently view by anyone,
even if you do have a bought domain?! Don't kid yourself; the fact of the matter is, you're
just one infantesimle speck in the universe of much greater websites than yours. Don't worry,
though - I got your back. Here, I'm gonna explain what meta tags are, and what they're used
for. I'm also going to show how to utilize them for your site, to increa....
My Theory Of Betting
what can make you stop? (10) i have just starting betting in the bookies, and found out a whole new knowledge and experiance.
winning makes you fell good and want to bet more to get more, or losing it makes you want to win it
back. but heres the question. how do you stop these bets? nothing seems to stop you from doing it.
maybe its in our human nature to be greedy and feel good. is there a limit to feel good in betting?
or do you always end up crying when you lose it all. knowledge is most important in betting (i
think) to work out the odds they are based on pass experiences. but if it wins all....
Div Tags Vs Tables For Layout
what should I use? (10) Hello everyone. First of all, hello. I want to make a really nice website. I don't want to use
tables but if I want to make a page with Adobe Image Ready it would be tables based. I know when I
use DIV tags it's simpler and the loading time seems to be faster. I don't know if i can
make a good looking site with just DIV tags. Does anyone have a preference when doing the layout of
their site? If I use DIV tags the main graphic on the site will be the banner. I'm not a very
big fan of heavy graphics. The content of the site is actually my main concern, and....
Robots Meta Tag Introduction
bots and spiders crawling the web Part 2 (5) Robots meta tags, do we need them. What good do they do and what does it control. “Robots Meta Tag”
are used to control if you want a spider or a bot to index a html page or not. You can give
permission to index your whole site and the spider will crawl all your pages. This is a great way
to control bots and spiders if you don’t have access to the root directory and robots.txt file.
Some search engines (not all) fully obey the “Robots Meta Tag”. What is the format of “Robots Meta
Tag” and where do I put it on my site? The “Robots Meta Tag” are placed in your HTML d....
Bad_pool_header
Blue screen Win Xp stop (11) Bad pool header Hi, do anyone know how to solve a BAD_POLL_HEADER... this is not a permanent error,
it comes and goes.. The only unsual thing that are new on this pc are: *SP2 upgrade (i think this
is the problem) and e-mule recently installed... Win XP starts normally, and when we try to login
into an user account POW.. The Killer blue screen appears... the messegae is long and in spanich..
so the relevant info BAD_POOL HEADER_ and 0x00000019....
Stop AUTO_INCREMENT From Jumping
(13) One of my MySQL tables has an ID column - it's a primary key with auto_increment. And it works.
If I add six rows they number 1 to 6 nicely. But if I delete rows 4 to 6 then add a new row, it
becomes row 7. And if I delete that it becomes row 8. How can I make it so the next auto_increment
is the smallest possible? (i.e. 1,2,3,4,5,6 - I delete 4-6 and the next row inserted becomes 4)....
Stop anything from making it ur homepage!
(52) I found out that you can override any homepage setting in Internet Options. If you have spyware, or
come accross it frequently, then this is a very good tweak for you. 1. Right-click on the IE icon
on your desktop. 2. Select Properties 3. Now in the "Target" box you will see "C:\Program
Files\Internet Explorer\IEXPLORE.EXE" Now by adding the url of the site to the end of this it
overrides any Homepage setting in internet options. ex: "C:\Program Files\Internet
Explorer\IEXPLORE.EXE" www.sdgamerz.tk Note: This works only with a custom shortcut to Internet
Explor....
Looking for meta, tags, stop, spiders
|
*SIMILAR VIDEOS*
Searching Video's for meta, tags, stop, spiders
|
advertisement
|
|
