Loading...


bookmark - Make Your Own Search Engine? Is it possible to make your own engine?

Make Your Own Search Engine? - Is it possible to make your own engine?

 
 Discussion by infamousflame with 30 Replies.
 Last Update: April 12, 2012, 3:40 pm
 
bookmark - Make Your Own Search Engine? Is it possible to make your own engine?  
Quickly Post to Make Your Own Search Engine? Is it possible to make your own engine? w/o signup Share Info about Make Your Own Search Engine? Is it possible to make your own engine? using Facebook, Twitter etc. email your friend about Make Your Own Search Engine? Is it possible to make your own engine? Print
Reply / Comment New Discussion / Topic Share / Bookmark E-Mail a Friend Print

Does anyone know how to make your own search engine? Does it take lots of time and careful programming or is it quite quick and easy to make a basic one? Post here!











   Sun Jun 19, 2005    Reply         

Youcould start by trying this link and maybe you can decide if that is what you want to do! How Intertnet Search Engines Work! Hope the link will get you going where you want to go.

   Sun Jun 19, 2005    Reply         

Just opnions.

I have never made a search engine though. Never had the need to. But I have some idea. It mostly depends on the search space. It less abouit programming and more about using the right tool. Searching Algorithms are no doubt one of the most studied algos out there with varying implementations..so is the case with search engines them selves. It would be wise to look into already developed search engines.

I need not mention sites like sourceforge.net tigris.com etc which host projects like these. If you find some thing interesting you may want to join them.

just curious , do you want to venture into search engine technology just for the sake of it or do u have an actual problem to solve (ie you are building a website of some sort)

Also search engines are taking new approaches these days , ie sound based , video , 3d etc etc and not just plain old text.

You may want to check http://en.wikipedia.org/wiki/Search_Engine out

   Sun Jun 19, 2005    Reply         


I was just curious about how easy it would be to make your own search engines , I don't think I have time to do it now though. It's probably all too technical for me. I don't know so much about programming, so I might not be able to do it. I know you said it isn't so much about programming, but it would still probably be too much of an extra workload for me, I've got a lot going on with my site and with my forum. Thanks for givng me such good info guys. :lol:

   Sun Jun 19, 2005    Reply         

QUOTE (infamousflame)

I was just curious about how easy it would be to make your own search engines , I don't think I have time to do it now though. It's probably all too technical for me. I don't know so much about programming, so I might not be able to do it. I know you said it isn't so much about programming, but it would still probably be too much of an extra workload for me, I've got a lot going on with my site and with my forum. Thanks for givng me such good info guys. :D
[post="40200"]<{POST_SNAPBACK}>[/post]


Creating a search engine needs a huge database i presume and a large brain to scour through all the algos of searching, sorting and implementing the best to suite your needs... I feel its about a effecient programming than about large programming. You need to have all the knowledge about internet and how it works. How you rate pages and index them for better searh...
i think you should start creating searchengine for a localhost now and make it a big boom on the internet if you succeed in implementing the better algos. Like google does...

Best of luck dude

   Tue Aug 9, 2005    Reply         

Programming the search engine is one thing - hardware is another! You need much space (many many harddisks), much RAM, many processors and voltage ... and that all costs MONEY. :D

GreetingZ

   Tue Aug 9, 2005    Reply         


Creating search engine is simple. Really! But if you want search engine like Google and yahoo so its not simple anymore. lol!

There's a lot of free search engine scripts out there ready to use for your site.

   Fri Sep 2, 2005    Reply         

Yes it's true

   Sat Sep 2, 2006    Reply         

QUOTE (infamousflame)


Does anyone know how to make your own search engine? Does it take lots of time and careful programming or is it quite quick and easy to make a basic one? Post here!

Link: view Post: 40168


Best practice to learn some topic is to explore existing implementations. Start by reading sources.
I can recommend MnoGoSearch and AspSeek. First written in perl AFAIK, second C++ and MySQL.
Happy hacking!

   Fri Nov 17, 2006    Reply         

If you would like to create a search section/site search area for your site, there are 3 methods:

1. Use free, easy to use, PHP scripts. You can find plenty here. Just find the free ones underneath the commercial ones and choose the one that is most suitable.

2. Use Google SiteSearch.

3. Investigate and research how to build your own PHP/ASP files to create your own search engine to index your site. You may also want a script for a sitemap.

The first two options are easiest. Good luck! :P

   Sun Nov 26, 2006    Reply         

Think of an unorganized library, with all the books lying here and there. I tell you to find me a book that contains any description of Neural Networks. To make matter worse, all the books have their indexes ripped apart. All that helps you is the title of the book and the contents.

Now just imagine what searching for a topic would be like. Your first step will be to search by the title of the books. But its not surprising that many books have weird titles which may not convey what the book might be containing within.

So in this case having a index in each book would have helped a load. But still, since the library is unorganized and books are lying in a pile, you would have to search the books by picking them up one by one. In the worst case, the book you are finding might be at the last of the heap. In this case the algorithm time complexity is n, where n is the no. of books in the library.

Now what you do is sort the books in sections depending upon their core theme. A book on artificial neural networks shall actually be stashed in Computer Science section and not Neurology/Biology section.

Now further more, to make things even more fast, you introduce something called a frequently issued section, that contain only the books frequently issued. It happens that most of the books requested for issue come from the frequent section. Simply, now you end up making your algorithm faster.

Now remember that you categorized books in sections? Let's do one more thing. The books that appear the most in bibliography of other books reserve a seat for themselves in the top shelf, indicating that they are the most popular and reliable books. This, my dear friend is your introduction to page rank algorithm used by Google.

Why I gave this example is because the very first model described in para 1 can be made in 2 hours. The time required and programming skills needed subsequently increase.

Now lets talk about programming and requirements.
Hardware (you may also do with a single PC with a 160GB HDD):
> Loads of storage space (RAID)
> A well connected database
> Multiple Servers for load sharing

Programming:
> Web crawler (that fetches the web pages and puts them in the Data warehouse)
> Data organizer (that arranges the data so that it can be mined efficiently)
> Search Algorithm

You can read some good white papers regarding search engine technology and settle upon a time frame in which you can make a search engine.

   Mon Nov 27, 2006    Reply         

making a search engine is a difficult and a verymuch financially expensive one ,if you want to make a real search engine like google hehe, the main thing is the crawlers who crawl billions of pages and the data they store in the servers ,big and pwerfull servers are needed to store and retrieve the data , making search engine for a particular site or forum is relatively easy ,just need to use the cron jobs to scan your site at regular interval for scanning all pages and make it available easily in the searches whenever someone search for it

   Wed Feb 7, 2007    Reply         

Not so much about programming? And how exactly would a search engine exist without programming, it would fall flat on it's arse.

The reverse however, may not work as intended, but it will do something. So i'd be willing to bet that programming is far more important.

Creating some form of Search Engine, you don't need to bother reading the garbage that everyone else opinionates on.

You need to think about what you need to be searching and how, start off with something programmed basic and just improve and built it from there.

You certainly won't have the "World OS Final" running from the get go if you know what I mean ;).

   Sat Feb 10, 2007    Reply         

I think Stumbleupon is doing something interesting by using their stumblers to help rate their sites. People enjoy using the plug-in and community and at the same time are contributing to their search engine.

   Sat Feb 10, 2007    Reply         

@foolakadugie, agreed. StumbleUpon definitely has an interesting system there. I like how often I get an interesting website everything I randomly browse around. It's better than typing in random words to Google and hoping for something interesting on my more boring days.

More on topic: Making a search engine is infinitely hard, especially if you want a successful search engine and not just something that's there for playing purposes. As simple as Google may look, that is only on the outside. If you really want something as successful as Google, it'll take a lot of work, just like CaptainRon has said.

On the other hand, you could also just make a small-scaled version of a search engine for fun so that you understand the basics of how it works. But if you were seriously dedicated, it's time to start setting up a schedule for yourself.

   Mon Apr 9, 2007    Reply         

Making a search engine would require a vast database and a very powerful processor to handle the large amounts of requests. I don't guess it'd take much hard disk space but the Processing power needed would be astronomical, weather it be one central data server or different nodes the combined Processing power would have to be very large. Unless it was a small, limited database and then I'd call it something else.

   Mon Apr 9, 2007    Reply         

If your going to make a Search Engine

Your going to need a big database engine and a fast server...

Alot of brains and you have to know other languages to make other sites like german ect...

Your gonna need alot of money to advertise it for a start.

   Mon Apr 9, 2007    Reply         

You'll have to do the same as everybody else really, start from scratch with something simple and then build it up from there.

It is quite possible to build a very complex and stable search engine without the need for ridiculous amounts of space and processing power (to begin with atleast).

You just got to make sure you can scale it well afterwards lol.

   Tue Apr 10, 2007    Reply         

well you could also try some of the softwares available which create a search engine for you:

Zoom Search
Search Engine Builder

   Sat May 12, 2007    Reply         

Look at my search engine here

I've made it using Google "Custom Search Engine". It works pretty well. To this moment I've indexed near 2000 sites.

http://www.google.com/coop/cse/

   Mon Jun 11, 2007    Reply         

I would suggest using pre-made PHP scripts. Ofcourse, that does not mean that you will not require heavy duty servers! <_<

   Wed Jun 13, 2007    Reply         

I started building my Search in 1999.
Been debating Desktop search for some time at:
http://channel9.msdn.com/showuserthreads.aspx?userid=31672

It's the only program I really use.

It evolved to do all the search related features I could think of.

   Fri Jun 22, 2007    Reply         

I have seen some script on the web some time ago, I think it is not hard to get the script and start the search engine but indexing webpages needs a lot of hardware. Another issue is how do you market your search engine and benefit from it as there are som many free search engines already availible for free?

But, if you make a search engine which can index and serve results for specific topic for example "golf tour search engine" maybe it can still grab a share in the market. Making general search engine and getting benefit from that is out off the question.

   Sun Oct 7, 2007    Reply         

Out of curiosity, does anyone have a website who provide a already made search engine, maybe the Python or JavaScript for it?

   Sun Nov 25, 2007    Reply         

Your own search web site could use existing search engines, and some kind of mask or skins, with customizable features to allow the user select between one or another, even several search engines, and how It will show the data in one web page.

If you are trying to create a search engine from scratch, you should begin creating your robo-spider first.

   Thu Jan 3, 2008    Reply         

I have a problemMake Your Own Search Engine?

I have a problem I can't figure out how to connect to my database I post the script below can any one tell me whats wrong with like coding errors

mysql_connect('localhost','root','root');My_select_db('SearchEngine');$GET = mysql("Select * From keywords");While ($getrow = mysql_fetch_assoc($get)){   $keywords =$getrow;  echo $keywords,'<br>';  }

 

-reply by Programer

 

   Sat Nov 20, 2010    Reply         

Very interesting topic but I am really disappointed to find no post which could be really helpful. Learning to make my own search engine will be really helpful for me (and for many other webmaster like me who want to implement their own search engine into their website). There are such solutions available which discourage forming your own search engine. Services like Google custom search engine are really easy to obtain and highly customizable so people seldom go for the effort of designing their own search engine. However, I must say that If I could make my own search engine, why would I go for an ad supported engine given to me by a third party.
My own search engine would really make my website look cleaner and I can use it in whatever way I want. Though the existing free services like google custom search engine are also very customizable but they still won't be able to compete with my own creation.
So I would love to hear from someone about how to make my own search engine for my website.

   Thu Feb 10, 2011    Reply         

Go to shinysearch.com

   Wed Apr 11, 2012    Reply         

Sphider is one search script that i know which allows meta search operations. It indexes and filters content based on custom code. I have seen some variations of that scripts that is hosted on some sites.


Another thing that was usable- yahoo search boss. I am not sure if it is working anymore. As yahoo is merged with bing, i guess the boss api may not be working anymore. Duckduckgo is the search engine that uses it.


The thing about search engine is that it is necessary to buid spam index and filters before bot wildely indexes everything. So it all depends on lots of coding, better hosting and diskspace and we're talking about datacenter here.

   Thu Apr 12, 2012    Reply         

As a newbie, I have had similar questions but I will give you one simple tip from my personal experience. Before anything else, you should assess your requirements and then spend your time accordingly. And if you are really into web-development industry, then I would recommend you to learn php and mysql. Once you get command over these two things, you would be able to do most of the things you see on the web, no matter whether it is a search engine or a simple website,.

   Thu Apr 12, 2012    Reply         

As for database, i am not sure it is going to work with mysql database. Google and other search engines are using something different which is likely to be custom coded.


Google app engine which hosts many apps with search feature makes use of something custom. The syntax for accessing database is quite something different. It doesnt look like sql to me.


Couch db, nosql and some file system friendly search engines are in use. I tried python,yaml and app engine for connecting two dbs. I thought about making it look like search engine and it didnt worked for me.

   Thu Apr 12, 2012    Reply         

Quickly Post to Make Your Own Search Engine? Is it possible to make your own engine? w/o signup Share Info about Make Your Own Search Engine? Is it possible to make your own engine? using Facebook, Twitter etc. email your friend about Make Your Own Search Engine? Is it possible to make your own engine? Print
Reply / Comment New Discussion / Topic Share / Bookmark E-Mail a Friend Print

Similar Topics:

Search Engine Optimization

Well, You have made a site and you got the data. You have put it online and want people to see it. The best and easiest way to get people to your site is Search Engines. But most of us know the vast amount of Competition on Internet. So here are some steps which will Definately let you rank better t ...more

   23-Jun-2005    Reply         

Yahoo Puts Other Search Engines To ...

As anti-google as I am, I cannot possible begin to say that google is the worst search engine ever. However, I can say that Hotbot is. I cannot stand Hotbot! I am also very curious as to what you all think is the worst search engine ever. And don't say yahoo! ...more

   07-Jul-2005    Reply         

Profusion.com The Original Meta Se...

I know all of you are busy using (and enjoying) Google's SE as your favourite one, but you would "think again" after reading this post... ...more

   31-May-2006    Reply         

Netbux.org Get paid to search the net!!!   Netbux.org Get paid to search the net!!! (4) (1) A9   A9