Grub (search engine)
Encyclopedia
Grub is an open source distributed search
crawler platform.
Users of Grub can download the peer-to-peer
grubclient software
and let it run during computer idle time. The client indexes the URLs
and send them back to the main grub server
in a highly compressed
form. The collective crawl could then, in theory, be utilized by an indexing system, such as the one being proposed at Wikia Search
. Grub is able to quickly build a large snapshot by asking thousands of clients to crawl and analyze a small portion of the web each.
Wikia has now released the entire Grub package under an open source software license
. However, the old Grub clients are not functional anymore. New clients can be found on the Wikia wiki.
. Undetermined copyright, patent or trademark rights from Grub, Inc. were purchased in 2003 for $1.3 million by LookSmart, Ltd. For a short time the original team continued working on the project, releasing several new versions of the software, albeit under a closed license.
Operations of Grub were shut down in late 2005. The site was reactivated on July 27, 2007, and the site is currently being updated. The original developers are assisting with the new deployment, and investigating the robots.txt issue, to ensure a repeat performance does not occur.
On July 27, 2007 Jimmy Wales
announced that Wikia
, Inc., the for-profit company developing the open source search engine Wikia Search
, had acquired Grub from LookSmart
. The cost was $50,000.
Distributed web crawling
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web crawling. Such systems may allow for users to voluntarily offer their own computing and bandwidth resources towards crawling web pages...
crawler platform.
Users of Grub can download the peer-to-peer
Peer-to-peer
Peer-to-peer computing or networking is a distributed application architecture that partitions tasks or workloads among peers. Peers are equally privileged, equipotent participants in the application...
grubclient software
Computer software
Computer software, or just software, is a collection of computer programs and related data that provide the instructions for telling a computer what to do and how to do it....
and let it run during computer idle time. The client indexes the URLs
Uniform Resource Locator
In computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
and send them back to the main grub server
Server (computing)
In the context of client-server architecture, a server is a computer program running to serve the requests of other programs, the "clients". Thus, the "server" performs some computational task on behalf of "clients"...
in a highly compressed
Bandwidth compression
In telecommunication, the term bandwidth compression has the following meanings:*The reduction of the bandwidth needed to transmit a given amount of data in a given time.*The reduction of the time needed to transmit a given amount of data in a given bandwidth....
form. The collective crawl could then, in theory, be utilized by an indexing system, such as the one being proposed at Wikia Search
Wikia Search
Wikia Search was a short-lived free and open-source Web search engine launched by Wikia, a for-profit wiki-hosting company founded in late 2004 by Jimmy Wales and Angela Beesley....
. Grub is able to quickly build a large snapshot by asking thousands of clients to crawl and analyze a small portion of the web each.
Wikia has now released the entire Grub package under an open source software license
Open-source license
An open-source license is a copyright license for computer software that makes the source code available for everyone to use. This allows end users to review and modify the source code for their own customization and/or troubleshooting needs...
. However, the old Grub clients are not functional anymore. New clients can be found on the Wikia wiki.
History
The project was started in 2000 by Kord Campbell, Igor Stojanovski, and Ledio Ago in Oklahoma CityOklahoma city
Oklahoma City is the capital and largest city of the U.S. state of Oklahoma.Oklahoma City may also refer to:*Oklahoma City metropolitan area*Downtown Oklahoma City*Uptown Oklahoma City*Oklahoma City bombing*Oklahoma City National Memorial...
. Undetermined copyright, patent or trademark rights from Grub, Inc. were purchased in 2003 for $1.3 million by LookSmart, Ltd. For a short time the original team continued working on the project, releasing several new versions of the software, albeit under a closed license.
Operations of Grub were shut down in late 2005. The site was reactivated on July 27, 2007, and the site is currently being updated. The original developers are assisting with the new deployment, and investigating the robots.txt issue, to ensure a repeat performance does not occur.
On July 27, 2007 Jimmy Wales
Jimmy Wales
Jimmy Donal "Jimbo" Wales is an American Internet entrepreneur best known as a co-founder and promoter of the online non-profit encyclopedia Wikipedia and the Wikia company....
announced that Wikia
Wikia
Wikia is a free web hosting service for wikis . It is normally free of charge for readers and editors, deriving most of its income from advertising, and publishes all user-provided text under copyleft licenses. Wikia hosts several hundred thousand wikis using the open-source wiki software MediaWiki...
, Inc., the for-profit company developing the open source search engine Wikia Search
Wikia Search
Wikia Search was a short-lived free and open-source Web search engine launched by Wikia, a for-profit wiki-hosting company founded in late 2004 by Jimmy Wales and Angela Beesley....
, had acquired Grub from LookSmart
LookSmart
LookSmart is an online advertising company based in San Francisco. LookSmart provides search advertising products and services to text advertisers, as well as targeted pay-per-click search and contextual advertising via its Search Advertising Network...
. The cost was $50,000.