Archived

This topic is now archived and is closed to further replies.

Would a neural network work for natural language proccessing?

This topic is 5279 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I was thinking of inputting ech character of what the user sais into the neural net and having it output an intellegent response, but I''m not sure how well this would work. Any advice?


-~-The Cow of Darkness-~- If you see the image I am online

Share this post


Link to post
Share on other sites
Guest Anonymous Poster
Speaking as someone quite knowledgeable about natural language processing (NLP), a simple neural net isn''t even going to produce acceptable outputs without being unbearably complex. Moreover, it''s a task that cannot be accomplished even with current cutting-edge NLP technology.

You may want to consider somehow limiting the range of possible inputs the player can produce, because otherwise the scope of the problem is so great as to make it intractable.

Anthony Serrano

Share this post


Link to post
Share on other sites
Without a clearer goal to aim for, I fear you''re not going to achieve anything. A neural net just maps inputs to outputs and you''re going to need to think carefully about how you''re classifying the inputs, what potential outputs you have, and how to judge the quality of the chosen outputs in order to train the net.

[ MSVC Fixes | STL Docs | SDL | Game AI | Sockets | C++ Faq Lite | Boost
Asking Questions | Organising code files | My stuff | Tiny XML | STLPort]

Share this post


Link to post
Share on other sites
I second Kylotan''s comment. Try something simpler.

If you haven''t already read them, try the tutorials at my website, they should help your understanding.






ai-junkie.com

Share this post


Link to post
Share on other sites
It''s a final no for me. I just do not believe in all these "let''s do neural networks to simulate how the brain gets intelligent".
To look at the best stuff done in this area (interpretation, memorisation, action), search the web for dmap, raps, mops, case based reasoning...

http://tfpsly.planet-d.net/english/ai.html
http://www.iit.edu/~sarimar/ai/index.html

Share this post


Link to post
Share on other sites
Hmm... You all still make it seem like I''m actually expecting it to work. Matt, yes, I already have read tutorials at your website and am also in possesion of your book "AI Techniques for Game Development." (It''s wonderful) Kylotan, I am not planning to achieve anything, just get a better understanding of how neural nets respond to input and generalize.

Share this post


Link to post
Share on other sites
Then try a similar problem that works a lot better with neural networks... that of (optical) character recognition (OCR). ANNs are very useful for this task and would give you good experience at applying them to a real world problem!

Cheers,

Timkin

[edited by - Timkin on June 4, 2003 10:45:10 PM]

Share this post


Link to post
Share on other sites
Yeah, instead of giving him a question and expecting an answer, give him a written question and have him output the question in a string

BTW: Can ANN''s work the other way round? From some general data (eg a 2x2 image) create a detailed output (eg a 16x16 image)? If it''s possible, it would be a nice lossy compression algo to store heightmaps or something like that (given that the ANN data is smaller than all the maps it can remember)

Sander Maréchal
[Lone Wolves Game Development][RoboBlast][Articles][GD Emporium][Webdesign][E-mail]


GSACP: GameDev Society Against Crap Posting
To join: Put these lines in your signature and don''t post crap!

Share this post


Link to post
Share on other sites
NNs are really good for fuzzy pattern matching. In a NLP situation, I''d say that you would have a tokenizer (typical procedural code), maybe a parser to try to generate a set of possible parse trees if the input is weird (or words are misspelled), then a neural net to deem which possible tree is the best choice.

NNs take strange input and give you a strange output. The simpler you can get them, the better they are. It''s best to use them as small parts of a large problem, using good procedural code to do everything that can be done that way.

NNs also can''t remember too much without getting big... the heightfield idea would work up to a point... you''d probably only be able to hold 8 maps total before it started outputting complete randomness...

Share this post


Link to post
Share on other sites
Can you use an ANN to do NLP? Short answer: no

If you want to parse an incoming sentence, you either need to parse the sentence (with a chartparser or standard rule-based parser, ala compiler design) or use statistics (learned markov models and bigram analyzers)

If you want to understand a sentence, that''s an entirely different beast as their is no meaning in a sentence, words are just pointers to mental constructs in your head. So to understand language you need to have the ideas already in your head, figure out how to access them and then map those to words

My personal opinion on so-called "neural nets" is that they''re worthless and are popular for no other reason than their name. i''ve written whole diatribes about this but the quick answer is to understand what a computer ANN is. Given some booleans/0-1 numbers, it multiplies numbers to come up with another number between 0 and 1 (normally a boolean). Anything you can define as a series of booleans giving rise to another boolean you can do in an ANN. But it''s probably faster and more accurate to use a decision tree

Any time someone tells you a neural network can compute any function, try having it add 2 and 2

(no, you cannot map to binary; if you do, i change the program to add any two arbitrary numbers)

(and obviously i''m only talking about FF/BP ANNs; other types like LeNet feature extractors and Boltzman machines are completely different beasts and not nearly as popular)

Ain''t i a stinker

-baylor, the anti-neural network crusader

Share this post


Link to post
Share on other sites
Please explain why you necroed this threa just to tell me that my idea won''t work (which has already been said many times, and I''m still going to try it) and explain wrongly how neural nets work? Lets see you use a decision tree to figure out wheather an image contains a human face. Not so cocky anymore, now are we?




-~-The Cow of Darkness-~-

If you see the image I am online

Check out PGNet.tk

Share this post


Link to post
Share on other sites
quote:
Original post by cowsarenotevil
Please explain why you necroed this threa just to tell me that my idea won''t work



Um, i don''t know what necroed means

The short answer to "why did you say NNs won''t work" is because they get recommended an awful lot and i think people who are given that advice don''t know the limitations of the technique. As to why i answered your question, it''s because no one (i think) had mentioned to look at chartparsing and markov models for NLP and i figured you''d like the pointers to some NLP information

quote:

explain wrongly how neural nets work?



i don''t think i did. Was their something specific that was wrong? Since others will read these posts, it''s worth getting the right information out there

quote:

Lets see you use a decision tree to figure out wheather an image contains a human face.



Not sure what you''re asking. Are you attempting to say that a decision tree can''t recognize faces? Then the quick answer is "yup, they can". Or do you really want to find a facial recognition system based on an DT? i don''t know any pointers to any because i think they''re generally done in other ways. Stuff like matching to eigenfaces. If you''re really interested in facial recognition i could ask for you - we have a good facial recognition program here

But again, of course you can figure out what is a face with a decision tree. But if you wanted to do it the human way, you wouldn''t use an ANN or decision tree or any other single-purpose algorithm. You''d do lots of processing in a special facial recognition module. You''d be looking for curves, lines, distance between features, feature location, etc. But if you had all that it''d be pretty trivial to autogenerate that with a DT algorithm like CART or C5.0

quote:

Not so cocky anymore, now are we?



Oh don''t be silly, i''m always cocky. Maybe not right, but that never stopped me from being cocky

Since you''re going to try an ANN approach to some type of language task, best of luck. Hopefully, whether it works or not, you''ll post your findings so others can learn from them

-b, who for the first time in his life has finally been called "cocky" - hurray!

Share this post


Link to post
Share on other sites
This is dead wrong. Neural networks are not in any way based on booleans (which is what makes them work well, in my opinion). You can input chars, floats, or anything, and they then can output anything.
quote:

Given some booleans/0-1 numbers, it multiplies numbers to come up with another number between 0 and 1 (normally a boolean).



And also, I meant making a program that can recognize specific faces. A decision tree would seem to be impossible, or am I missing something? You'd have to run a whole bunch of if statements on each pixel, where as with a well designed neural net, you'd input a face, and then use backpropogation or similar to teach it to match the image to the right ID of the human.

EDIT: Oh, and "Necroed" stands for "necromanced," meaning brought back from the dead (i.e. replying to an old thread)




-~-The Cow of Darkness-~-

If you see the image I am online

Check out PGNet.tk

[edited by - cowsarenotevil on June 30, 2003 1:38:19 AM]

Share this post


Link to post
Share on other sites
quote:
Original post by baylor
Um, i don''t know what necroed means



The practice of dredging up a long-since-dead thread...

quote:
Original post by baylor
As to why i answered your question, it''s because no one (i think) had mentioned to look at chartparsing and markov models for NLP and i figured you''d like the pointers to some NLP information



I think the problem here is that the original poster was considering using NLP as a testbed for ANNs, which as has already been stated above, a bad idea for learning about and testing ANNs. Certainly the methods you suggest are better for NLP...

... to further this, if cowsarenotevil is thinking of doing NLP with some other means, I can recommend the work of Kevin Murphy on using Hidden Markov Models for NLP. Slightly different from the traditional MM approach, but having sufficiently better results as to warrant an investigation.

Timkin

Share this post


Link to post
Share on other sites
timkin, the sole purpose for attempting to use a neural net to respond to language, is to see what happens when you try to make a neural net respond to language. I''m not trying to test neural nets in general, I just want to experiment. I got the answer to my question: no. But why should I not experiment with it anyway? Maybe I''ll even make it do something useful.




-~-The Cow of Darkness-~-

If you see the image I am online

Check out PGNet.tk

Share this post


Link to post
Share on other sites
quote:
Original post by cowsarenotevil
timkin, the sole purpose for attempting to use a neural net to respond to language, is to see what happens when you try to make a neural net respond to language.



By all means, go ahead and try. I certainly am not the sort of person to try and stifle someones ideas.

quote:
Original post by cowsarenotevil
But why should I not experiment with it anyway?



Perhaps because more experienced people have suggested that it won''t work, based on an understanding of ANNs and NLP and the mismatch between the two...???

quote:
Original post by cowsarenotevil
Maybe I''ll even make it do something useful.



Many innovative discoveries came from people ''bucking the trend'' and ''ignoring the nay-sayers''... by all means, try it and see. Who knows, you might prove us all wrong and I''ll happily eat my hat if you do!

Cheers,

Timkin

Share this post


Link to post
Share on other sites
quote:
Original post by cowsarenotevil
Neural networks are not in any way based on booleans (which is what makes them work well, in my opinion). You can input chars, floats, or anything, and they then can output anything.


You can input them and make them output anything based on numbers. I do not think words, or worst ideas, can be thought about as numbers.

Of course you could give each word a specific identifying number, but how would you represent the meanings of the words and how they interact ?

I think that's an old dream that never became true. (personal thought)And I'm quite sure it will never become true.(/personal thought)

[edited by - TfpSly on June 30, 2003 9:18:10 AM]

Share this post


Link to post
Share on other sites
Ah Baylor, the crusader from comp.game.ai. I''ve read some of your threads with interest, you occasionally have some interesting angles, but you do tend to fire from the hip in a valiant effort to wind everyone up. For example, you wrote a long diatribe about how useless A* is to game programmers... without really knowing how A* works or how it is applied in games, or even how fast it is compared to other search methods. You were not even aware of the proof that A* is guaranteed to find the shortest path given an appropriate heuristic.

I am responding to your comments because you seem to be doing the same with ANNs. You appear to have learned a little about them and are now making enormous assumptions based upon that limited amount of information. Exactly the sort of thing you dislike when you say "because they get recommended an awful lot and i think people who are given that advice don''t know the limitations of the technique."

Your description of what an ANN is, and what it can or cannot do, is telling. Most ANNs do not use Booleans and the input or output from an ANN can be any real value, although it is true to say that the majority of feedforward networks use normalized inputs because this makes for easier training/learning.

You go on to say

"Any time someone tells you a neural network can compute any function, try having it add 2 and 2

(no, you cannot map to binary; if you do, i change the program to add any two arbitrary numbers)"


What makes you think an ANN cannot add two numbers? Why do you think that mapping to binary makes it a simpler task? Given a range of integer values, a neural net can be created that will add any two of them together. (input in binary or decimal, it makes no difference) An ANN could do the same with real numbers also, but it would take a hell of a lot of training to get the error levels down!

In addition, you have this to say about face recognition:

"But if you wanted to do it the human way, you wouldn''t use an ANN or decision tree or any other single-purpose algorithm. You''d do lots of processing in a special facial recognition module. You''d be looking for curves, lines, distance between features, feature location, etc."

How do you think ANNs are trained to recognize faces? If you want an ANN to catagorize something then it is common practice to use some form of feature extraction. ANNs can be extremely good at catagorization based upon incomplete or fuzzy data (compared to other methods), that''s why they are used so much in commercial ''thing'' recognition software.

You are correct of course that ANNs get recommended far too much by people who don''t really know them. (You are living proof of the converse). Unfortunately ANNs are a ''buzz'' technology, and as such are likely to get promoted and rubbished far too often. This is just human nature. ANNs do have their uses though. Certainly their ability to generalize and to learn is a very powerful feature. However, they should be used as just another tool in the AI programmers work belt (just like your favorite, the decision tree) and not as some sort of magical panacea to all a game''s AI problems.


My Website: ai-junkie.com | My Book: AI Techniques for Game Programming

Share this post


Link to post
Share on other sites