Joe Lau's wiki: Main/Connectionism History
Readings
- A brief discussion by Jack Copeland
- Pollack, J. B. (1989). Connectionism: Past, Present, and Future. Artificial Intelligence Review, 3, 3-20. http://www.demo.cs.brandeis.edu/papers/nnhistory.pdf
- James A. Anderson and Edward Rosenfeld (eds.) Talking Nets hkulib:b1899492
Three stages in the development of connectionism
- Associationism in philosophy (up till about 1940)
- The beginning of neural computation and the death of the perceptron (1943-1969)
- The return of connectionism (1970-)
Stage 1. Early empiricist associationism in philosophy
@For in a discourse of our present civil war, what could seem more impertinent than to ask, as one did, what was the value of a Roman penny? Yet the coherence to me was manifest enough. For the thought of the war introduced the thought of the delivering up the King to his enemies; the thought of that brought in the thought of the delivering up of Christ; and that again the thought of the 30 pence, which was the price of that treason: and thence easily followed that malicious question; and all this in a moment of time, for thought is quick.^^^Hobbes (1588-1679)@
@To me, there appear to be only three principles of connexion among ideas, namely, Resemblance, Contiguity in time or place, and Cause or Effect.^^^David Hume@
Stage 2. The beginning of neural computation
- McCulloch and Pitts (1943). A logical calculus of ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5: 115-133.
- Information: http://www.mind.ilstu.edu/curriculum/modOverview.php?modGUI=212
@Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.^^^Hebb, D.O. (1949). The Organization of Behavior@
- F. Rosenblatt. (1958). The Perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65:386-408.
Total input = T = w1x1+ w2x2 + ... + wn xn
Output = f(T)
Hype:
@NEW NAVY DEVICE LEARNS BY DOING; Psychologist Shows Embryo of Computer Designed to Read and Grow Wiser. WASHINGTON, July 7 (UPI) -- The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.^^^New York Times@
Devastating critique and setback
Easy for perceptrons:
| input 1 | input 2 | output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
Impossible for perceptrons (XOR):
| input 1 | input 2 | output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
- Minsky and Papert (1969). Perceptrons. Cambridge, Mass.: MIT Press. isbn:0262631113
- Perceptrons cannot solve linearly-nonseparable problems, e.g. XOR
Stage 3. The return of connectionism
Connectionism makes a comeback.
- Multi-layer networks overcome the limitations of perceptrons
- More powerful learning algorithms
- Suggestive psychological applications
- David E. Rumelhart, James L. McClelland, The PDP Research Group (1986). Parallel Distributed Processing, Vol. 1, Explorations in the Microstructure of Cognition : Foundations . Cambridge, Mass.: MIT Press. isbn:0262181207
- James L. McClelland, David E. Rumelhart, The PDP Research Group (1987). Parallel Distributed Processing, Vol. 2, Psychological and Biological Models. Cambridge, Mass.: MIT Press. isbn:0262631105