Make Your Own Search Engine? - Is it possible to make your own engine?

Think of an unorganized library, with all the books lying here and there. I tell you to find me a book that contains any description of Neural Networks. To make matter worse, all the books have their indexes ripped apart. All that helps you is the title of the book and the contents.

Now just imagine what searching for a topic would be like. Your first step will be to search by the title of the books. But its not surprising that many books have weird titles which may not convey what the book might be containing within.

So in this case having a index in each book would have helped a load. But still, since the library is unorganized and books are lying in a pile, you would have to search the books by picking them up one by one. In the worst case, the book you are finding might be at the last of the heap. In this case the algorithm time complexity is n, where n is the no. of books in the library.

Now what you do is sort the books in sections depending upon their core theme. A book on artificial neural networks shall actually be stashed in Computer Science section and not Neurology/Biology section.

Now further more, to make things even more fast, you introduce something called a frequently issued section, that contain only the books frequently issued. It happens that most of the books requested for issue come from the frequent section. Simply, now you end up making your algorithm faster.

Now remember that you categorized books in sections? Let's do one more thing. The books that appear the most in bibliography of other books reserve a seat for themselves in the top shelf, indicating that they are the most popular and reliable books. This, my dear friend is your introduction to page rank algorithm used by Google.

Why I gave this example is because the very first model described in para 1 can be made in 2 hours. The time required and programming skills needed subsequently increase.

Now lets talk about programming and requirements.
Hardware (you may also do with a single PC with a 160GB HDD):
> Loads of storage space (RAID)
> A well connected database
> Multiple Servers for load sharing

Programming:
> Web crawler (that fetches the web pages and puts them in the Data warehouse)
> Data organizer (that arranges the data so that it can be mined efficiently)
> Search Algorithm

You can read some good white papers regarding search engine technology and settle upon a time frame in which you can make a search engine.

 

 

 

Comment/Reply (w/o sign-up)

making a search engine is a difficult and a verymuch financially expensive one ,if you want to make a real search engine like google hehe, the main thing is the crawlers who crawl billions of pages and the data they store in the servers ,big and pwerfull servers are needed to store and retrieve the data , making search engine for a particular site or forum is relatively easy ,just need to use the cron jobs to scan your site at regular interval for scanning all pages and make it available easily in the searches whenever someone search for it

Comment/Reply (w/o sign-up)

Not so much about programming? And how exactly would a search engine exist without programming, it would fall flat on it's arse.

The reverse however, may not work as intended, but it will do something. So i'd be willing to bet that programming is far more important.

Creating some form of Search Engine, you don't need to bother reading the garbage that everyone else opinionates on.

You need to think about what you need to be searching and how, start off with something programmed basic and just improve and built it from there.

You certainly won't have the "World OS Final" running from the get go if you know what I mean .

Comment/Reply (w/o sign-up)

I think Stumbleupon is doing something interesting by using their stumblers to help rate their sites. People enjoy using the plug-in and community and at the same time are contributing to their search engine.

Comment/Reply (w/o sign-up)

@foolakadugie, agreed. StumbleUpon definitely has an interesting system there. I like how often I get an interesting website everything I randomly browse around. It's better than typing in random words to Google and hoping for something interesting on my more boring days.

More on topic: Making a search engine is infinitely hard, especially if you want a successful search engine and not just something that's there for playing purposes. As simple as Google may look, that is only on the outside. If you really want something as successful as Google, it'll take a lot of work, just like CaptainRon has said.

On the other hand, you could also just make a small-scaled version of a search engine for fun so that you understand the basics of how it works. But if you were seriously dedicated, it's time to start setting up a schedule for yourself.

Comment/Reply (w/o sign-up)

Making a search engine would require a vast database and a very powerful processor to handle the large amounts of requests. I don't guess it'd take much hard disk space but the Processing power needed would be astronomical, weather it be one central data server or different nodes the combined Processing power would have to be very large. Unless it was a small, limited database and then I'd call it something else.

Comment/Reply (w/o sign-up)

If your going to make a Search Engine

Your going to need a big database engine and a fast server...

Alot of brains and you have to know other languages to make other sites like german ect...

Your gonna need alot of money to advertise it for a start.

Comment/Reply (w/o sign-up)

You'll have to do the same as everybody else really, start from scratch with something simple and then build it up from there.

It is quite possible to build a very complex and stable search engine without the need for ridiculous amounts of space and processing power (to begin with atleast).

You just got to make sure you can scale it well afterwards lol.

Comment/Reply (w/o sign-up)

well you could also try some of the softwares available which create a search engine for you:

Zoom Search
Search Engine Builder

Comment/Reply (w/o sign-up)

Look at my search engine here

I've made it using Google "Custom Search Engine". It works pretty well. To this moment I've indexed near 2000 sites.

http://www.google.com/coop/cse/

Comment/Reply (w/o sign-up)