Hello! While browsing the net today I found this great resource related to web robots, search engines, web crawlers, spiders and such stuff. The page is: The web robots You can find many usefull stuff there if you are webmaster, or you are interested in making your own spider/indexer, or similar stuff. You can even look at their database of Web Robots - some of them even have their own source code, so you can compile one for yourself...

Reply

Where would you put your robot once you have it? I can't find that information on the site. Would you put it on a server (like astahost) or on your own computer? And does indexing take a lot of space and bandwidth?

Reply

I think it takes alot of space... and consumes bandwith and much of processor time...

Reply

Oh. Well, then, I'd better just have Google index my site for me, instead of wearing out Astahost's servers. Nice link, though.

Reply

Anyway many of that WEB-robots are capable to catch emails, and harvest them into database... spammers use that very often, to get targeted audience. I figured if we research some of this methods, maybe we could better protect ourself from spam and junk emails... Anyway, I'll post what I discover later here..

Reply

google robots are the best robots and it has like hundreds of them , it uses a lot of bandwith so it isn`t good for little sites , they are good for sites like google , msn etc

Reply

well idea of this is testing and learning, anyway if you know how many spiders work, you will be able to make your web pages better and that way increase your page rank on search engines - and that's what we all want

Reply

the link is good, but not much of my use. i will try to see how to make it in use for futur..
for the time being i will let google do the work for me ;-)

Reply

I think that, with the current state of search engine optimization, we should let the SEs do the indexing themselves. Unless one knows exactly how every single SE works to index and rank their site, one will most likely hurt his ranking at one engine or another.

I remember one time when Google used H1 and TITLE tags as a primary criteria for their ranking, while MSN had them at 4th and 6th. And right now, Google mainly uses a system called Vector Analysis, where they analyse the overall theme of your Website, and adjust your ranking accordingly. It's still work-in-progress, but Google partly uses it, while not many others do.

So my point is, until SEs can reach a certain level of standardization, we should let each SE do what it likes most. Plus, it doesn't really take that much bandwidth, not more than any hungry visitor to your site would take.

Reply

When you say "theme", do you mean the content theme or the visual theme of the site? And how can the robot adjust the ranking with the H1, title or theme? I don't get how a computer can tell what is good and what is bad.

Reply