Welcome Guest ( Log In | Register )



3 Pages V  < 1 2 3 >  
Reply to this topicStart new topic
> Make Your Own Search Engine?, Is it possible to make your own engine?
CaptainRon
post Nov 27 2006, 06:24 AM
Post #11


Premium Member
Group Icon

Group: Members
Posts: 238
Joined: 9-September 05
Member No.: 8,400



Think of an unorganized library, with all the books lying here and there. I tell you to find me a book that contains any description of Neural Networks. To make matter worse, all the books have their indexes ripped apart. All that helps you is the title of the book and the contents.

Now just imagine what searching for a topic would be like. Your first step will be to search by the title of the books. But its not surprising that many books have weird titles which may not convey what the book might be containing within.

So in this case having a index in each book would have helped a load. But still, since the library is unorganized and books are lying in a pile, you would have to search the books by picking them up one by one. In the worst case, the book you are finding might be at the last of the heap. In this case the algorithm time complexity is n, where n is the no. of books in the library.

Now what you do is sort the books in sections depending upon their core theme. A book on artificial neural networks shall actually be stashed in Computer Science section and not Neurology/Biology section.

Now further more, to make things even more fast, you introduce something called a frequently issued section, that contain only the books frequently issued. It happens that most of the books requested for issue come from the frequent section. Simply, now you end up making your algorithm faster.

Now remember that you categorized books in sections? Let's do one more thing. The books that appear the most in bibliography of other books reserve a seat for themselves in the top shelf, indicating that they are the most popular and reliable books. This, my dear friend is your introduction to page rank algorithm used by Google.

Why I gave this example is because the very first model described in para 1 can be made in 2 hours. The time required and programming skills needed subsequently increase.

Now lets talk about programming and requirements.
Hardware (you may also do with a single PC with a 160GB HDD):
> Loads of storage space (RAID)
> A well connected database
> Multiple Servers for load sharing

Programming:
> Web crawler (that fetches the web pages and puts them in the Data warehouse)
> Data organizer (that arranges the data so that it can be mined efficiently)
> Search Algorithm

You can read some good white papers regarding search engine technology and settle upon a time frame in which you can make a search engine.
Go to the top of the page
 
+Quote Post
richie
post Feb 7 2007, 11:37 AM
Post #12


Member - Active Contributor
Group Icon

Group: Members
Posts: 98
Joined: 16-December 06
Member No.: 18,408



making a search engine is a difficult and a verymuch financially expensive one ,if you want to make a real search engine like google hehe, the main thing is the crawlers who crawl billions of pages and the data they store in the servers ,big and pwerfull servers are needed to store and retrieve the data , making search engine for a particular site or forum is relatively easy ,just need to use the cron jobs to scan your site at regular interval for scanning all pages and make it available easily in the searches whenever someone search for it
Go to the top of the page
 
+Quote Post
Chesso
post Feb 10 2007, 07:21 AM
Post #13


Teh Coder
Group Icon

Group: Members
Posts: 1,053
Joined: 18-April 06
From: Australia
Member No.: 12,833



Not so much about programming? And how exactly would a search engine exist without programming, it would fall flat on it's arse.

The reverse however, may not work as intended, but it will do something. So i'd be willing to bet that programming is far more important.

Creating some form of Search Engine, you don't need to bother reading the garbage that everyone else opinionates on.

You need to think about what you need to be searching and how, start off with something programmed basic and just improve and built it from there.

You certainly won't have the "World OS Final" running from the get go if you know what I mean tongue.gif.
Go to the top of the page
 
+Quote Post
foolakadugie
post Feb 10 2007, 08:07 AM
Post #14


Premium Member
Group Icon

Group: Members
Posts: 241
Joined: 4-August 05
From: Los Angeles
Member No.: 7,624



I think Stumbleupon is doing something interesting by using their stumblers to help rate their sites. People enjoy using the plug-in and community and at the same time are contributing to their search engine.
Go to the top of the page
 
+Quote Post
Arbitrary
post Apr 9 2007, 08:56 PM
Post #15


Premium Member
Group Icon

Group: [HOSTED]
Posts: 337
Joined: 17-June 06
From: Adblock life
Member No.: 13,992



@foolakadugie, agreed. StumbleUpon definitely has an interesting system there. I like how often I get an interesting website everything I randomly browse around. It's better than typing in random words to Google and hoping for something interesting on my more boring days.

More on topic: Making a search engine is infinitely hard, especially if you want a successful search engine and not just something that's there for playing purposes. As simple as Google may look, that is only on the outside. If you really want something as successful as Google, it'll take a lot of work, just like CaptainRon has said.

On the other hand, you could also just make a small-scaled version of a search engine for fun so that you understand the basics of how it works. But if you were seriously dedicated, it's time to start setting up a schedule for yourself.
Go to the top of the page
 
+Quote Post
SilverFox
post Apr 9 2007, 09:29 PM
Post #16


Premium Member
Group Icon

Group: Members
Posts: 206
Joined: 26-February 07
From: Texas
Member No.: 20,598



Making a search engine would require a vast database and a very powerful processor to handle the large amounts of requests. I don't guess it'd take much hard disk space but the Processing power needed would be astronomical, weather it be one central data server or different nodes the combined Processing power would have to be very large. Unless it was a small, limited database and then I'd call it something else.
Go to the top of the page
 
+Quote Post
Rs2cheaters
post Apr 9 2007, 09:50 PM
Post #17


Newbie [ Level 2 ]
Group Icon

Group: Members
Posts: 14
Joined: 9-April 07
Member No.: 21,375



If your going to make a Search Engine

Your going to need a big database engine and a fast server...

Alot of brains and you have to know other languages to make other sites like german ect...

Your gonna need alot of money to advertise it for a start.

Go to the top of the page
 
+Quote Post
Chesso
post Apr 10 2007, 01:28 AM
Post #18


Teh Coder
Group Icon

Group: Members
Posts: 1,053
Joined: 18-April 06
From: Australia
Member No.: 12,833



You'll have to do the same as everybody else really, start from scratch with something simple and then build it up from there.

It is quite possible to build a very complex and stable search engine without the need for ridiculous amounts of space and processing power (to begin with atleast).

You just got to make sure you can scale it well afterwards lol.
Go to the top of the page
 
+Quote Post
webguru
post May 12 2007, 04:00 PM
Post #19


Newbie [ Level 2 ]
Group Icon

Group: Members
Posts: 13
Joined: 12-May 07
Member No.: 21,909



well you could also try some of the softwares available which create a search engine for you:

Zoom Search
Search Engine Builder
Go to the top of the page
 
+Quote Post
translector
post Jun 11 2007, 01:30 PM
Post #20


Newbie [ Level 2 ]
Group Icon

Group: Members
Posts: 14
Joined: 11-June 07
Member No.: 22,557




Look at my search engine here

I've made it using Google "Custom Search Engine". It works pretty well. To this moment I've indexed near 2000 sites.

http://www.google.com/coop/cse/
Go to the top of the page
 
+Quote Post

3 Pages V  < 1 2 3 >
Reply to this topicStart new topic

Collapse

> Similar Topics

Topics Topics
  1. Add A Search Box To My Web Site(10)
  2. Enhanced Audio Cd's(2)
  3. How Do I Make A RPG Battle System?(13)
  4. Tekken 1-5(16)
  5. Online Multiplayer Game(11)
  6. Make Your Own FPS Game(31)
  7. Photoshop Tutorial: How To Make That Popular Three Pixel Border(7)
  8. How Search Engines Operate(6)
  9. Remove Window Animation - Make It Run Faster(8)
  10. Photoshop Tutorial: How To Make A Userbar For Signatures(31)
  11. Blingo Search Engine(3)
  12. How To Make A CS 1.6 Map ?(13)
  13. How To Run A Proxy On a Web-Server?(18)
  14. Lock Workstation Shortcut(8)
  15. How To Make Realistic Fire!(13)
  1. New Text Based Game Maker(12)
  2. A Beginner Wanting To Learn How To Make A Web Game(4)
  3. Graphcal User Interfaces In Java(4)
  4. Powerpoint Based Games!(14)
  5. Access(17)
  6. Do Google Search Better Than Yahoo?(14)
  7. Love Calculator(1)
  8. How To: Make A Simple Php Forum System(4)
  9. Spider Simulator Tool(0)
  10. Can't Make This Query Working.(2)
  11. Make It Impossible To View Page Source(10)
  12. Get Paid To Search Yahoo!(10)
  13. Mysql Storage Engine Error 28(5)


 



- Lo-Fi Version Time is now: 6th July 2008 - 09:36 AM