Original Post
1) Introduction Forgive the length of this thread. I decided to organize it into sections for easier reading. I would like to do image classification with neural networks. For this, I know that feed-forward neural networks can be used. Even single-layer ones (perceptron-like). The problem I am running into is that my classification is going to be quite complex (eg: recognizing the same object from different perspectives) and I anticipate that this would take a neural feed-forward network several layers. 2) My problem Now, just to analize a 64x64 black and white image (that's 4096 pixels), I might need say, 5 layers. The problem is that, from what I know, in a standard feed-forward network, each neuron is linked to all neurons of the successive layer. Hence, if I have just two layers, there will be 4096 x 4096 links (over 16 million links). This is obviously pretty bad, because the number of links is exponential. Now if I want to add colors (which would be important in my problem), I need 3 times more input neurons, so 12288 on the first level. But if I have 2 levels of that, this makes 150M links! Assuming each link can be compressed into 32 bits (which is somewhat inconvenient), this makes for over 600MB of link data, just for a two layer neural network to analyze my color image. But even there, I would probably want more than two layers, and more definition than 64x64. If I wanted to support RGB color 256x256 images, I would have 196608 input neurons. If I had a second layer with that many neurons, this means 3.87 x 10^10 links. More than my modest 2GB of RAM can possibly hope to hold. 3) Neural networks using less memory So now, I've been thinking about this, and there isn't much hope for this approach. If I used the disk, this would still be painfully slow and complex. But then I thought... This is not even how the human brain works. In our brain, neurons are not organized in layers, nor do they link to all other neurons in their proximity. I read in an article that some of our neurons were linked to as much as about 4000 nearby neurons, but most probably are not! So I began to wonder, would there be a way to build a layered (for simplicity) neural network where neurons are only linked to a small subset of the subsequent layer. Say, each neuron has at most 1024 links? This way, with an input matrix of size 196608 and a second layer with the same number of neurons, I have 196608x1024 links = 201 M links, which is big, but still manageable. If I could work with even less links, say 64 links per neuron, then this would be 196608x64 links, so 12.6M links, which is very manageable... And assuming 8 bytes per link, would still only be ~100 MB of data, possibly allowing me as much as 15 layers and 3 million neurons (realistically). The neurons could even spread their links around, so that a neuron is linked to all the neurons "facing" it on the next layer, but also to a few farther neurons, to allow the important information to propagate faster. The obvious question is, has this been done? Are there neural networks organized so that neurons only link with the nearby ones, so they can have more neurons and take less memory. What about learning algorithms for those? Would the regular back-propagation algorithm still work, or would new ones need to be implemented? 4) Time-dependent neural networks? I am also interested in knowing if there are neural networks that do not work like the typical feed-forward network. That is, typically, neural networks work in a static time frame. The input is fed, the computation is performed (signals travel instantly), and the output is obtained. However, are there artificial neural networks that work on a frame-based mechanism. I imagined something like this, which I feel is more organic and perhaps closer to the way our brain tissue works: You feed input signals every frame to some selected input neurons, and neurons only fire if their threshold is reached. Until that happens, they keep accumulating charge. Each frame, a neuron can send discharges to the ones it is immediately connected to, but the signals do not travel multiple levels. The network can also have feedback loops in its organization, where a neuron can send a signal, and indirectly trigger another signal being sent to itself from elsewhere... Which means that the processing does not necessarily ever stop. Output may periodically be output from some selected output neurons. This ANN would have the advantage that it could have "memory" of past events, and process things continuously (such as ongoing sounds). It would not be restricted to processing static input samples. The main problem is that I have no idea how such an ANN would be trained, apart from something inefficient like this (which seems very inefficient): 1) Reset ANN (neutralize still ongoing signals, but preserve weights) 2) Perform random variations to ANN weights 3) Feed training input set into ANN 4) Measure and rate performance again, if improved, keep modifications, otherwise reset to previous weight configuration 5) Neutralize still ongoing signals while preserving weights 6) Goto #1