Jump to content
  • Advertisement
  • entries
    628
  • comments
    1454
  • views
    1010665

The One Search Google Can't Do

Sign in to follow this  
ApochPiQ

180 views

I believe that searching is a hard problem. Although today the term "Google" is virtually synonymous with search, we still tend to obscure and ignore the real difficulties of information retrieval. In fact, Ask Jeeves has recently been exploiting some of this for marketing purposes, although they have fallen short of truly addressing the root issue.

The root issue is, in a nutshell, that we don't understand searching.


Sure, we can correlate keywords to web pages and retrieve them at astounding rates. We can attach all kinds of sophisticated importance to various bits of HTML and certain sequences of words. We can even detect synonyms and expand query results based on tangentially-related words. Many of these things are areas which Google has mastered, if not pioneered outright. They have done this so effectively that "search" and "Google" are fundamentally inextricable to many people. In fact, it is increasingly common to instruct people seeking knowledge to "just Google it". The unwritten expectation is that anyone can find anything on Google - or, as loath as we may be to admit it, comparable search engines.

But this is a delusion, a fantasy that exists almost exclusively on the Internet. Roll back the clock 15 years and see what "search" meant: for the most part, it meant learning to use indexes, cross references, bibliographies, footnotes, and the Dewey Decimal System. It meant hard work, often correlating concepts which seem only tenuously related, making leaps of intuition or sometimes blind faith from one book/subject/author to the next. Search, or as it was more properly called then, research, carried a connotation of challenge, difficulty, real work. Research was hard, and we knew it. We knew it so well that it was considered a fundamental part of higher-level academic learning - what one researched was rarely so important as the fact that one learned to research at all.


Back in the present, we have lost touch with the notion of research. Today, it largely means "spending a couple minutes on Google". If one is deadly serious, one might spend a few whole hours, following links two or three levels deep. The truly determined (or perhaps only the truly anachronistic) will trudge off to the library and poke around in books for a while. Tellingly, however, many people determine which books to check via - you guessed it - searching the Internet.

I've commented before on the form of confirmation bias the Internet tends to create. In general, the Internet gives us access to a tremendous amount of information and stuff; but how much of it do we truly exploit? With the growing consciousness that the Internet is not necessarily 100% truth, many people will reject things they find on the Web offhand if they disagree. This is especially visible during those times when politics is discussed loudly and at length by just about everybody.

In the comments on that entry, I touched on the larger issue - it's bad enough that the Internet lets us feed our biases while pretending we're broadening our views, but the real problem is that truly broading one's views is an unsolved search problem.


This is fundamentally connected to the concept of education as led by a more learned/experienced teacher. The entire notion of mentorship and apprenticeship is bound deeply to this problem. Unfortunately, our society has largely forgotten how those concepts work; without them, the Internet lacks much of its potential to spread enlightenment to the human race.

The challenge can be summed up neatly with a simple question: if you need to learn about something, but don't even know the words to use to describe it, how do you find it on Google? Suppose there's some niche field of science that you don't even know exists; how can you find its existence?

Answer: you don't. If you're ridiculously lucky, you can play with synonyms and hope someone has asked a similar question on a forum or newsgroup, and thereby acquire the "proper" keywords to solve your question. Most of the time, you'll end up with nothing, or at least nothing that really helps. Now you're toast, especially if you don't have a library card.


Stated simply, there is one search that Google (or any other search mechanism that exists, for that matter) cannot do: it can't search for something we are totally unaware of. More subtly, it cannot answer the questions we are unable to ask.

The solution, of course, is to introduce people. Discussion and dissemination of knowledge will eventually bring the answer to light. It is by finding and dialoguing with a large number of people that we expand our knowledge; keyword-based searches can only give us additional depth, not additional breadth. Yet breadth is precisely what we need to achieve a rich and effective outlook on life.


For a time in history, this wasn't an issue; the sum total of human knowledge (or at least the set of all known and active fields) could be communicated in a few books or by a good teacher. It was in this atmosphere that institutions of higher learning became prevalent in Western societies. Yet in the intervening centuries, we've developed such a mass of information that even decades of specialization and careful study can leave a person unaware of vast tracts of knowledge directly pertinent to their own chosen field, to say nothing of fields they haven't sought degrees in.




For someone like myself, who thrives on exploring the metamind of humanity, this is a vexing issue. Clearly the most efficient answer would be to seek out people as different from oneself as possible, and absorb as much as possible from them. This assumes two things; the easy assumption is that such people will be willing to share their knowledge. This is not such a problem, since people are fundamentally (for the most part) enthralled at the chance to be narcicisstic for a bit and ramble about themselves. (Witness the pervasiveness of blogging.)

The larger assumption is that one has a way to find such people in the first place. In the old days, those with an autodidactic bent would travel the world, seek the learned men, philosophers, and teachers, and learn what they could. Today's environment is simultaneously more and infinitely less conducive to such a lifestyle. While the professional student is by no means extinct, it is a difficult road, at least in much of the world - the U.S. in particular.

Worse, it conflicts with other demands on a person's time. If one has a job and other responsibilities which cannot (or should not) readily be ignored, it is infeasible to travel the world to pick the brains of other people. In rare cases where such a thing is possible, it relies unduly on the availability of those other people - and such availability is by no means guaranteed.


The Internet seemingly provides the perfect remedy; we can communicate at leisure with people from across the globe, with minimal constraints and inconveniences. We can even see and hear each other thanks to VOIP and video chat technologies. Yet the convenience of communication is moot given the difficulty of connection; the Internet is governed by Google's flavour of search, which as we've already seen is not exactly conducive to exploring outside our boundaries.


I believe the idea of finding people and concepts (as opposed to factoids and keywords) is severely underdeveloped in today's Internet infrastructure. I suspect that the societal changes we've seen arising from the Internet are trivialities compared to the true potential of a global communications network that also lets us all probe far beyond our areas of expertise.

The adage says that we can't learn something unless we almost already know it. Perhaps it is unfounded idealism, a sort of rebellion against such a cynical notion, but I for one am not prepared to accept this as a universal truth. Rather, it appears to me to be a mere artifact of how we have gone about structuring - and retrieving - our knowledge as a human race.


But as much as I dislike the current state of affairs, I'm at a loss to envision anything better. I'd look for solutions, but that's one search Google can't do.
Sign in to follow this  


3 Comments


Recommended Comments

Touche. Today's mentality of "just google for it and you're done" is a real problem and has a potential risk of limiting many people's views to just a few selected pages, while missing all the paralell information the search engine doesn't show you. But I don't rely much on Google to learn the more academic materials; I usually look it up on Wikipedia. Unlike Google, it also expands your search on breadth, not just depth -- you should've mentioned that in your post :) Then, there are also search engines specialized on finding academic papers (www.b-on.pt, for example, but the search site is in Portuguese). You can expand the breadth of your search a lot by following a few references in the papers you find. In a nutshell, these are my defences against the common trap that Google has turned itself into, and until now they've been doing fine :)

Share this comment


Link to comment
Quote:
Original post by ApochPiQ
I believe that searching is a hard problem. Although today the term "Google" is virtually synonymous with search, we still tend to obscure and ignore the real difficulties of information retrieval. In fact, Ask Jeeves has recently been exploiting some of this for marketing purposes, although they have fallen short of truly addressing the root issue.

The root issue is, in a nutshell, that we don't understand searching.


Sure, we can correlate keywords to web pages and retrieve them at astounding rates. We can attach all kinds of sophisticated importance to various bits of HTML and certain sequences of words. We can even detect synonyms and expand query results based on tangentially-related words. Many of these things are areas which Google has mastered, if not pioneered outright. They have done this so effectively that "search" and "Google" are fundamentally inextricable to many people. In fact, it is increasingly common to instruct people seeking knowledge to "just Google it". The unwritten expectation is that anyone can find anything on Google - or, as loath as we may be to admit it, comparable search engines.

But this is a delusion, a fantasy that exists almost exclusively on the Internet. Roll back the clock 15 years and see what "search" meant: for the most part, it meant learning to use indexes, cross references, bibliographies, footnotes, and the Dewey Decimal System. It meant hard work, often correlating concepts which seem only tenuously related, making leaps of intuition or sometimes blind faith from one book/subject/author to the next. Search, or as it was more properly called then, research, carried a connotation of challenge, difficulty, real work. Research was hard, and we knew it. We knew it so well that it was considered a fundamental part of higher-level academic learning - what one researched was rarely so important as the fact that one learned to research at all.


Back in the present, we have lost touch with the notion of research. Today, it largely means "spending a couple minutes on Google". If one is deadly serious, one might spend a few whole hours, following links two or three levels deep. The truly determined (or perhaps only the truly anachronistic) will trudge off to the library and poke around in books for a while. Tellingly, however, many people determine which books to check via - you guessed it - searching the Internet.


You subtly changed the definition of 'search' to 'research', and I don't think that's a logically valid transition.

To put it in terms loosely related to yours, a ‘search’, in this context, is largely an act of requesting some pre-existing information by using some formation of words or information that loosely describes or relates to the information you are in search of.

‘Research’, however, involves picking points in the general region of a desired, pre-existing set of information (that part is the guess-work), and using bits of data to (hopefully) construct a trail of information to follow that leads you to your desired information. When researching to learn something new, you assemble the bits of data you find into various structures, until you have something that stands on its own.

Quote:

I've commented before on the form of confirmation bias the Internet tends to create. In general, the Internet gives us access to a tremendous amount of information and stuff; but how much of it do we truly exploit? With the growing consciousness that the Internet is not necessarily 100% truth, many people will reject things they find on the Web offhand if they disagree. This is especially visible during those times when politics is discussed loudly and at length by just about everybody.

In the comments on that entry, I touched on the larger issue - it's bad enough that the Internet lets us feed our biases while pretending we're broadening our views, but the real problemis that truly broading one's views is an unsolved search problem.


I think the method of research that I described above is possible using search engines and the internet, even if it is not as effective as traditional methods.

Quote:

This is fundamentally connected to the concept of education as led by a more learned/experienced teacher. The entire notion of mentorship and apprenticeship is bound deeply to this problem. Unfortunately, our society has largely forgotten how those concepts work; without them, the Internet lacks much of its potential to spread enlightenment to the human race.


I don't understand how apprenticeship can help enlighten the human race, or why a lack of apprenticeship means a lack of enlightenment. Is enlightenment synonymous with the broadening of views you spoke of earlier? If so, how does apprenticeship help broaden views?

Is this really related to the internet directly, or is it, as you say, a problem with our society (and, whose society are you speaking of? The World’s? Western culture's?)

Quote:

The challenge can be summed up neatly with a simple question: if you need to learn about something, but don't even know the words to use to describe it, how do you find it on Google? Suppose there's some niche field of science that you don't even know exists; how can you find its existence?


search for "Fields of science", in order to obtain a list of known fields of science? You don't have to search for the name of the thing you are looking for.

Quote:

Answer: you don't. If you're ridiculously lucky, you can play with synonyms and hope someone has asked a similar question on a forum or newsgroup, and thereby acquire the "proper" keywords to solve your question. Most of the time, you'll end up with nothing, or at least nothing that really helps. Now you're toast, especially if you don't have a library card.


How would you find that field of science in the library? Would it be more effective? Is it not possible on the internet?

Quote:

Stated simply, there is one search that Google (or any other search mechanism that exists, for that matter) cannot do: it can't search for something we are totally unaware of. More subtly, it cannot answer the questions we are una

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

Participate in the game development conversation and more when you create an account on GameDev.net!

Sign me up!