Earlier search engines held an index of a few hundred thousand pages and documents, and received maybe one or two thousand inquiries each day. Today, a top search engine will index hundreds of millions of pages, and respond to tens of millions of queries per day.
How are the pages indexed? How do the search routines really work? Only the operators of the search engines know for sure, and this is generally very closely held proprietary information. The process is continually improved, as all useful computer software is.
The information that the operators of the search engines will release, combined with the results of research into why certain pages are at the top of the list of results of certain searches, can help you out to gain good website optimization.
Using this information, you can make a reasonable determination of what techniques can be used to enhance a web site's visibility and effectiveness in the search engines.
Through an analysis of your existing or proposed web site, you can recommend and implement changes that can dramatically improve your site's chances of being found.