Connectionism makes a comeback.

• Multi-layer networks overcome the limitations of perceptrons
• More powerful learning algorithms
• Suggestive psychological applications

Impossible for perceptrons (XOR):

Multi-layer networks

http://philosophy.hku.hk/joelau/media/rsg1.jpg

https://upload.wikimedia.org/wikipedia/en/5/52/Mark_I_perceptron.jpeg

Output = f(T)

Total input = T = w₁x₁+ w₂x₂ + ... + w_n x_n

Easy for perceptrons:

Impossible for perceptrons:

The perceptron as a simple one-layer network (wikimedia)

https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/Perceptron.svg/456px-Perceptron.svg.png

• McCulloch and Pitts (1943). A logical calculus of ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5: 115-133.
• Information: http://www.mind.ilstu.edu/curriculum/modOverview.php?modGUI=212

@Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.^^^Hebb, D.O. (1949). The Organization of Behavior@

@Let us assume that the persistence or repetition of a reverberatory activity (or "trace") tends to induce lasting cellular changes that add to its stability.… When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.@

1. Associationism in philosophy (up till about 1940)
2. The beginning of neural computation and the death of the perceptron (1943-1969)

Stage 1. Early empiricist associationism in philosophy

Stage 2. The beginning of neural computation

Hype:

@NEW NAVY DEVICE LEARNS BY DOING; Psychologist Shows Embryo of Computer Designed to Read and Grow Wiser. WASHINGTON, July 7 (UPI) -- The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.^^^New York Times@

• Perceptrons cannot solve linearly-nonseparable problems, e.g. XOR

Stage 3. The return of connectionism

Readings

Three stages in the development of connectionism

Stage 1. Early empiricist associationism in philosophy (up till about 1940)

Stage 2. The beginning of connectionist computational models - the age of perceptrons (1943-1969)

Theories of the mechanisms of neural computation

The perceptron as a simple one-layer network

Devastating critique and setback

Empiricist associationism

@To me, there appear to be only three principles of connexion among ideas, namely, Resemblance, Contiguity in time or place, and Cause or Effect.^^^David Hume@

@For in a discourse of our present civil war, what could seem more impertinent than to ask, as one did, what was the value of a Roman penny? Yet the coherence to me was manifest enough. For the thought of the war introduced the thought of the delivering up the King to his enemies; the thought of that brought in the thought of the delivering up of Christ; and that again the thought of the 30 pence, which was the price of that treason: and thence easily followed that malicious question; and all this in a moment of time, for thought is quick.^^^Hobbes (1588-1679)@

• James A. Anderson and Edward Rosenfeld (eds.) Talking Nets hkulib:b1899492

• James A. Anderson and Edward Rosenfeld (eds.) Talking Nets hkulib:b1899492

• Hobbes (1588-1679)

@For in a discourse of our present civil war, what could seem more impertinent than to ask, as one did, what was the value of a Roman penny? Yet the coherence to me was manifest enough. For the thought of the war introduced the thought of the delivering up the King to his enemies; the thought of that brought in the thought of the delivering up of Christ; and that again the thought of the 30 pence, which was the price of that treason: and thence easily followed that malicious question; and all this in a moment of time, for thought is quick.@

@To me, there appear to be only three principles of connexion among ideas, namely, Resemblance, Contiguity in time or place, and Cause or Effect.@