Would a neural network work for natural language proccessing?

Started by
26 comments, last by cowsarenotevil 20 years, 9 months ago
quote:Original post by baylor
Um, i don''t know what necroed means


The practice of dredging up a long-since-dead thread...

quote:Original post by baylor
As to why i answered your question, it''s because no one (i think) had mentioned to look at chartparsing and markov models for NLP and i figured you''d like the pointers to some NLP information


I think the problem here is that the original poster was considering using NLP as a testbed for ANNs, which as has already been stated above, a bad idea for learning about and testing ANNs. Certainly the methods you suggest are better for NLP...

... to further this, if cowsarenotevil is thinking of doing NLP with some other means, I can recommend the work of Kevin Murphy on using Hidden Markov Models for NLP. Slightly different from the traditional MM approach, but having sufficiently better results as to warrant an investigation.

Timkin
Advertisement
timkin, the sole purpose for attempting to use a neural net to respond to language, is to see what happens when you try to make a neural net respond to language. I''m not trying to test neural nets in general, I just want to experiment. I got the answer to my question: no. But why should I not experiment with it anyway? Maybe I''ll even make it do something useful.



-~-The Cow of Darkness-~-

If you see the image I am online

Check out PGNet.tk
-~-The Cow of Darkness-~-
quote:Original post by cowsarenotevil
timkin, the sole purpose for attempting to use a neural net to respond to language, is to see what happens when you try to make a neural net respond to language.


By all means, go ahead and try. I certainly am not the sort of person to try and stifle someones ideas.

quote:Original post by cowsarenotevil
But why should I not experiment with it anyway?


Perhaps because more experienced people have suggested that it won''t work, based on an understanding of ANNs and NLP and the mismatch between the two...???

quote:Original post by cowsarenotevil
Maybe I''ll even make it do something useful.


Many innovative discoveries came from people ''bucking the trend'' and ''ignoring the nay-sayers''... by all means, try it and see. Who knows, you might prove us all wrong and I''ll happily eat my hat if you do!

Cheers,

Timkin
quote:Original post by cowsarenotevil
Neural networks are not in any way based on booleans (which is what makes them work well, in my opinion). You can input chars, floats, or anything, and they then can output anything.


You can input them and make them output anything based on numbers. I do not think words, or worst ideas, can be thought about as numbers.

Of course you could give each word a specific identifying number, but how would you represent the meanings of the words and how they interact ?

I think that's an old dream that never became true. (personal thought)And I'm quite sure it will never become true.(/personal thought)

[edited by - TfpSly on June 30, 2003 9:18:10 AM]
Ah Baylor, the crusader from comp.game.ai. I''ve read some of your threads with interest, you occasionally have some interesting angles, but you do tend to fire from the hip in a valiant effort to wind everyone up. For example, you wrote a long diatribe about how useless A* is to game programmers... without really knowing how A* works or how it is applied in games, or even how fast it is compared to other search methods. You were not even aware of the proof that A* is guaranteed to find the shortest path given an appropriate heuristic.

I am responding to your comments because you seem to be doing the same with ANNs. You appear to have learned a little about them and are now making enormous assumptions based upon that limited amount of information. Exactly the sort of thing you dislike when you say "because they get recommended an awful lot and i think people who are given that advice don''t know the limitations of the technique."

Your description of what an ANN is, and what it can or cannot do, is telling. Most ANNs do not use Booleans and the input or output from an ANN can be any real value, although it is true to say that the majority of feedforward networks use normalized inputs because this makes for easier training/learning.

You go on to say

"Any time someone tells you a neural network can compute any function, try having it add 2 and 2

(no, you cannot map to binary; if you do, i change the program to add any two arbitrary numbers)"


What makes you think an ANN cannot add two numbers? Why do you think that mapping to binary makes it a simpler task? Given a range of integer values, a neural net can be created that will add any two of them together. (input in binary or decimal, it makes no difference) An ANN could do the same with real numbers also, but it would take a hell of a lot of training to get the error levels down!

In addition, you have this to say about face recognition:

"But if you wanted to do it the human way, you wouldn''t use an ANN or decision tree or any other single-purpose algorithm. You''d do lots of processing in a special facial recognition module. You''d be looking for curves, lines, distance between features, feature location, etc."

How do you think ANNs are trained to recognize faces? If you want an ANN to catagorize something then it is common practice to use some form of feature extraction. ANNs can be extremely good at catagorization based upon incomplete or fuzzy data (compared to other methods), that''s why they are used so much in commercial ''thing'' recognition software.

You are correct of course that ANNs get recommended far too much by people who don''t really know them. (You are living proof of the converse). Unfortunately ANNs are a ''buzz'' technology, and as such are likely to get promoted and rubbished far too often. This is just human nature. ANNs do have their uses though. Certainly their ability to generalize and to learn is a very powerful feature. However, they should be used as just another tool in the AI programmers work belt (just like your favorite, the decision tree) and not as some sort of magical panacea to all a game''s AI problems.


My Website: ai-junkie.com | My Book: AI Techniques for Game Programming

Ok, instead of fighting each other, let''s return to the topic of the thread and try to give some constructive answer:

You want to try NLP with ANN. Ok, there can be several problems (as stated before), but let''s try to figure out how can you implement it:

1) First of all, ANN have a fixed number of inputs, if you want to enter sentences or words, you have a problem. Since a sentence has variable length. You should use "windowing", i.e.: if you use a 8-size window, the sentence "I am a student", should be analised using 8-character long windows: "I am a s", " am a st", "am a stu", "m a stud", " a stude", "a studen", " student".
Then make each of this windows be the input of your NN and then analise each one of the 7 outputs (one for each window).

2) But the main problem is data representation. Even you do windowing. Each input of a NN is a real number, so if you map characters to real numbers (for instance, by their ASCII value), the neural net will interpret that "a" is closer to a "b" than to an "e" (because their ASCII values are closer). I don''t know how do you plan to deal with this. Any way, with enough training, may be you can avoid this problem.

3) A NN is just a function approximator, so first of all, you need to define the function you plan to approximate. For instance, if you use 8-long windows (as I showed before), may be you can have a NN with 6 outputs. all the outputs are 0 by default, if the 1st output is 1, it''s because a NOUN has been detected, if the 2nd output is 1, it''s because a VERB has been detected, same for the rest with ARTICLES, ADJECTIVES, PREPOSITIONS and ADVERBS. But this is just the beginning of the processing, you need to define well the function you want to approximate in order to train your network.

I hope to have been of any help!
Is everyone ignoring the fact that the most common implementations of chatterbot are simple pattern matchers? ie input -> black box -> classification -> output.

If you had a stages to your system (some if not all built on NNs) -

1. identify important words ("the cat sat on the mat" gives "cat, sat, mat".

2. Maybe look up some knowledge on the words (are they verbs, animals, objects, people, events, etc)

3. Feed each into a NN which outputs a type of phrase to reply. (eg "cat" might output a "do you like .." question, or suggest that you produce some information about a the subject.)

4. Select a phrase, parse and output.

Note that 1, 2 are basically there to convert a string into some form of enumerated type and 4 is there to convert a set of numbers (possibly percetages for each possible answer type, possible a enum of question type) in to a string.

What you''d ideally do it allow the NN to get multiple words so that it would eventually be able to tell that "do you like cats" (all important words) should be responded to in a certain way (ie don''t answer a question with a question).

Personally I think that the prerequestite of a NN in this system is a design flaw. However, unlike most here I don''t think it''s entirely imcompatible with NL.

The best you are going to get with this probably some form of finite state machine type phrase type in -> phrase type out system. If you require an "intelligent" response then your system needs to be able to understand the word (eg "the mat" is a specific mat). However a NN chat bot could be possible (just want the world needs )
Kleene showed the equivalence of DFAs (deciders of regular langauges)and discrete recurrent ANNs way back in the 50s/60s. Of course regular langauges arent all that interesting, but if you let the activations of your recurrent net be continuous (and your numbers have infinite precision) any turing machine can be constructed. So by the above and the church-turing thesis, any of your other AI approaches can be run on a neural net

This topic is closed to new replies.

Advertisement