Readings

Distributed representations

• Classical cognitive architecture = a theory of mental processes that assumes LOT.
• Is connectionism an alternative to LOT?
• Two kinds of distributed representations - structured vs un-structured.

An alternative to LOT?

Pinker : two mechanisms - one for regular verbs ("talk") and one for irregulars ("go").
Plunkett, Rumelhart, ... : just one single network for both verb classes.

• Recall the three levels : the task level which describes what the system can do, the computational level which describes the representations and computations used, the hardware level which is about how the computations are realized and constructed.

Four possibilities

• Scenario 1 : Connectionism is a theory at the hardware level. It tells us how classical cognitive architectures are made out of neural machinery.
• Scenario 2 : Connectionism is a theory at the computational level. It tells us how classical computations are implemented by connectionist computations and representations at a lower computational level.
• Scenario 3 : Connectionism is a theory at the computational level, an alternative to the classical approach. It denies the existence of a computational level with a classical cognitive architecture.
• Scenario 4 : Connectionism is a theory at the computational level. It supplements classical theories to provide hybrid theories of the mind which make use of both classical and connectionist computations.

Comments

Re scenario #1,#2

• In most simulations, individual computing units do not seem to correspond to individual neurons.
• Many training rules are biologically unrealistic. (eg back propagation does not scale well, and cannot deal with one-shot learning.)
• Connectionist representations : localist vs distributed.
• Localist representation - one node for one meaning. Cannot deal with productivity.
• Structured, distributed representations can implement compositional representations - example

Re scenario #3

• Connectionists opt for networks that employ unstructured, distributed representations.
• These representations are not compositional, and hence not classical. The key issue is whether networks with such representations can explain systematicity and productivity.
• Some such networks are systematic and productive to a limited extent. But can they still perform satisfactorily when they are scaled up?

Distributed memory in Ramsey, W., Stich, S. P., & Garon, J. (1991)

• network architecture
• representations

Scenario #4

• Interesting alternative which is also under investigation.
• Can distributed representations deal with binding?

Defending the hybrid approach

• Simulation research shows that distributed representations are powerful and useful.
• But cognition cannot appeal only to unstructured distributed representations.
• Reason #1 : LOT needed to explain free transformation in central cognitive processes (e.g. conscious thoughts).
• Reason #2 : Networks that make use of unstructured distributed representations rely on structured distributed representations as inputs and outputs.
• An analogy with zip files.
• An example of a hybrid approach - LOT in working memory vs unstructured representations in long-term memory.