The mapping from language in the brain to the language of the brain

Abstract

Human language consists of at least three levels of representation, namely sound patterns, syntactic patterns, and semantic information. These three different levels of representation are acquired and stored in neocortical brain structures during the first years of life. They form the representational primitives at Marr’s computational level. During language processing, these representational primitives are retrieved and unified on-line (i.e. in real time and from left to right), to produce and comprehend the complex utterances that speakers and listeners are capable of exchanging. This requires a specification of Marr’s algorithmic level of analysis, here referred to as Unification. A system with the capacity to show complex language generation and interpretation has to meet these representational and algorithmic requirements. However, when it comes to neural implementations, it requires a mapping of representational and algorithmic levels onto the informational language of the brain itself. This mapping remains one of the major, and largely unmet challenges for a neurobiological account of human language.

As a way into this mapping problem, I will outline a computational approach to modeling language processing in spiking recurrent. Sequential input is non-linearly mapped into a high-dimensional neural state-space and the internal dynamics is subsequently decoded onto a set of read-out neurons using machine learning techniques. Read-outs are viewed as a measurement device to characterize the encoded information and provide a theory bridging between neuronal processes and concepts at computational and algorithmic levels. The approach is well-suited for testing the computational role of various neurobiological features, adaptation mechanisms, and network architectures. This will help to elucidate the role of (a) brain connectivity, (b) memory at various time-scales and (c) unsupervised, local learning and adaptation mechanisms supporting the language system's capacity to reconstruct hierarchically structured interpretations from sentence input. I will exemplify this approach within a neurobiologically motivated model which maps sentence input onto sequences of thematic roles and integrates these into sentence-level semantic representations.