-A Brief History of Search Engines

"How could the world beat a path to your door when the path was uncharted, uncatalogued, and could be discovered only serendipitously?" — Paul Gilster, Digital Literacy
The World Wide Web is different from anything we have known. Within the virtual reality of the Web, we can only see and hear things (at least at the time of this article). Because of this limitation, the Web forces us to find new ways to interact.
For instance, if I wanted to buy a book, I would go down to the local book store, select one I like, pay for it, and go home. The book store is usually in a visible place and has a sign out front, making it relatively easy to find.
But in cyberspace, there's no place to "turn." I have only my computer screen in front of me. Somehow, I need to find a place to purchase the book I want. There's no street on my screen so I can't drive around on the Web (I could "surf," but that's hit and miss; even then I still need to know where to start). Sometimes it's obvious: type in the name of the bookstore, add a .COM (as in barnesandnoble + .com) and it's a pretty good bet you're going to end up where you want to go. But what if it's a specialty bookstore and doesn't have a Web site with an obvious URL?
One solution to this problem is the search engine. In fact, it's probably one of the most widely used methods for navigating in cyberspace. Considering the amount of information that's available from a good search engine, it's similar to having the Yellow Pages, a guide book and a road map all-in-one.
Search engines can provide much more information than just the URL of a Web site. They can also locate reviews, help to compare prices, and even find if there have been any reported problems with the product or the manufacturer. Typing in "books" into the Google search engine returns about 9,270,000 results. If we refine the search to "books, Internet", we end up with about 6,070,000 results. We can narrow our search further to "books, Internet, search engines" and we will get about 803,000 results. If we know the book's author, let's say Danny Sullivan, then we enter "books, Internet, search engines, sullivan" and Google now returns about 10,900 results (of course, these results will change from day to day).
For many people, using search engines has become routine. Not bad for a technology that's not even 20 years old. But how did search engines come into being? What are the origins of this entity that prowls the outer reaches of cyberspace?
Note: This is by no means an exhaustive history of search engines. There's a resource list at the end of this article for more in-depth research.
The Early Beginnings of the Internet and the World Wide Web(or Your Tax Dollars at Work)
In 1957, after the U.S.S.R. launched Sputnik (the first artificial earth satellite), the United States created the Advanced Research Projects Agency (ARPA) as a part of the Department of Defense. Its purpose was to establish U.S. leadership in science and technology applicable to the military.
Part of ARPA's work was to prepare a plan for the United States to maintain control over its missiles and bombers after a nuclear attack. Through this work the ARPANET — a.k.a. the Internet — was born. The first ARPANET connections were made in 1969 and in October 1972 ARPANET went 'public.'
Almost 20 years after the creation of the Internet, the World Wide Web was born to allow the public exchange of information on a global basis. It was built on the backbone of the Internet.
According to Tim Berners-Lee, creator of the World Wide Web, "The Internet [Net] is a network of networks. Basically it is made from computers and cables.... The [World Wide] Web is an abstract imaginary space of information. On the Net, you find computers — on the Web, you find documents, sounds, videos, ... information. On the Net, the connections are cables between computers; on the Web, connections are hypertext links. The Web exists because of programs which communicate between computers on the Net. The Web could not be without the Net. The Web made the Net useful because people are really interested in information and don't really want to have [to] know about computers and cables."
With information being shared worldwide, there was eventually a need to find that information in an orderly manner.
Archie, Veronica, and Jughead(or The History of Search Engines Beginning at Riverdale High)
The very first tool used for searching on the Internet was called "Archie". (The name stands for "archives" without the "v", not the kid from the comics). It was created in 1990 by Alan Emtage, a student at McGill University in Montreal. The program downloaded the directory listings of all the files located on public anonymous FTP (File Transfer Protocol) sites, creating a searchable database of filenames.
While Archie indexed computer files, "Gopher" indexed plain text documents. Gopher was created in 1991 by Mark McCahill at the University of Minnesota. (The program was named after the school's mascot). Because these were text files, most of the Gopher sites became Web sites after the creation of the World Wide Web.
Two other programs, "Veronica" and "Jughead," searched the files stored in Gopher index systems. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives) provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead (Jonzy's Universal Gopher Hierarchy Excavation And Display) was a tool for obtaining menu information from various Gopher servers.
I, Robot
In 1993, MIT student Matthew Gray created what is considered the first robot, called World Wide Web Wanderer. It was initially used for counting Web servers to measure the size of the Web. The Wanderer ran monthly from 1993 to 1995. Later, it was used to obtain URLs, forming the first database of Web sites called Wandex.
According to The Web Robots FAQ, "A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. Web robots are sometimes referred to as web wanderers, web crawlers, or spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them."
Initially, the robots created a bit of controversy as they used large amounts of bandwidth, sometimes causing the servers to crash. The newer robots have been tweaked and are now used for building most search engine indexes.
In 1993, Martijn Koster created ALIWEB (Archie-Like Indexing of the Web). ALIWEB allowed users to submit their own pages to be indexed. According to Koster, "ALIWEB was a search engine based on automated meta-data collection, for the Web."
Enter the Accountants
Eventually, as it seemed that the Web might be profitable, investors started to get involved and search engines became big business.
Excite was introduced in 1993 by six Stanford University students. It used statistical analysis of word relationships to aid in the search process. Within a year, Excite was incorporated and went online in December 1995. Today it's a part of the AskJeeves company.
EINet Galaxy (Galaxy) was established in 1994 as part of the MCC Research Consortium at the University of Texas, in Austin. It was eventually purchased from the University and, after being transferred through several companies, is a separate corporation today. It was created as a directory, containing Gopher and telnet search features in addition to its Web search feature.
Jerry Yang and David Filo created Yahoo in 1994. It started out as a listing of their favorite Web sites. What made it different was that each entry, in addition to the URL, also had a description of the page. Within a year the two received funding and Yahoo, the corporation, was created.
Later in 1994, WebCrawler was introduced. It was the first full-text search engine on the Internet; the entire text of each page was indexed for the first time.
Lycos introduced relevance retrieval, prefix matching, and word proximity in 1994. It was a large search engine, indexing over 60 million documents in 1996; the largest of any search engine at the time. Like many of the other search engines, Lycos was created in a university atmosphere at Carnegie Mellon University by Dr. Michael Mauldin.
Infoseek went online in 1995. It didn't really bring anything new to the search engine scene. It is now owned by the Walt Disney Internet Group and the domain forwards to Go.com.
Alta Vista also began in 1995. It was the first search engine to allow natural language inquires and advanced searching techniques. It also provides a multimedia search for photos, music, and videos.
Inktomi started in 1996 at UC Berkeley. In June of 1999 Inktomi introduced a directory search engine powered by "concept induction" technology. "Concept induction," according to the company, "takes the experience of human analysis and applies the same habits to a computerized analysis of links, usage, and other patterns to determine which sites are most popular and the most productive." Inktomi was purchased by Yahoo in 2003.
AskJeeves and Northern Light were both launched in 1997.
Google was launched in 1997 by Sergey Brin and Larry Page as part of a research project at Stanford University. It uses inbound links to rank sites. In 1998 MSN Search and the Open Directory were also started. The Open Directory, according to its Web site, "is the largest, most comprehensive human-edited directory of the Web. It is constructed and maintained by a vast, global community of volunteer editors." It seeks to become the "definitive catalog of the Web." The entire directory is maintained by human input.