Posted by: Sleepless January 30, 2009
Login in to Rate this Post:
0 
? 
?
Thapap,
I can but give you a glimpse of what happens. I have no in-depth knowledge either, but i know where to go if i need to find them.
A generic search engine open source software that i used once was - ht-dig! It basically uses Digging, Merging and Searching techniques to crawl-index-search websites. Check out more at htdig.org
So the engine starts with crawling "SPIDER" that traverses across every links they find. Then, they maintain an 'index' of words, titles, headings or the page itself to the URL they traversed. Most of the times, webmasters have their Meta-Tags and keywords that help the spider to get their websites *crawled*. Webmasters can as well use Robots.txt file to deny crawling access to specific areas within their domain.
The win scenario for Sagoon:
Yahoo Boss! does it all within some parts of a second. The idea to present the user with what sagoon thinks the user is looking for - constitutes of all the efficiency - Sagoon would be looking forward to now. Eventually, they'd have their indexes and their complexities as they mention they have their own bots crawling around. Maintaining indexes, searching across with minimal possible overhead has many researches associated alongside. Like the Map-Reduce that Google uses for processing across large clusters of indexing servers.
I would, along with you, go to research.yahoo.com to read more about the search technologies further. But its not in my itinerary.