Smart Semiotics Webinar Series. Sensemaking across the human organism – October 23, 2022.

1 Comment

1. I was greatly heartened by the talk given on the 29th of April by Adolfo Garcia. It was an honour also to be counted among the panellists. I note that his project is trying to heal the rift between the two cultures of ‘personal understanding’ and ‘scientific understanding’. The ‘Two Cultures and the Scientific Revolution’ is a book by CP Snow first published in the ‘50s and it was quickly countered by literary critic FR Leavis, about the passion of ‘authentic’ experience, especially as manifested in English literature (1) . This translated for me into finding a proper connection between personal and scientific understandings, and the project was defined well by Levi Strauss as the reintegration of culture in nature in all its physical and chemical conditions. I believe that the uncompromising goal of seeking a grand theory of brain and behaviour was shared among us despite our diverse backgrounds.
   I have had a similar project for nearly 50 years, but the tool kit described by Adolofo had mostly not yet been invented. At last we get hard neurological evidence of the relationship between cortical process and a natural classification of the lexicon, in this case by action words..

   I also used a multi-dimensional approach in the ‘70s, comprising a hard impedance measure of fundamental frequency, two dimensional nose movement plots, phonetic analysis aligned precisely with the video record by visual inspection of plosives, and a transcription in natural language. I was studying the human face because I thought it would bring up all the methodological problems in studying behaviour, while at the same time having a well-defined physical subject matter. I recorded just half an hour of spontaneous chat, and arranged these modalities like a musical score, eight tracks in all with the vertical divisions at 1/10 second. It was a pure transcription, but I was aware that my 3 D nose movement plots lacked a 4th dimension. My subjects were trying to make sense to each other, but laughs, silences, extraneous noises in the immersive context were all part of the melody. The ‘control’ for this experiment was ‘uncontrol’, which felt dangerous and irresponsible in 1976, but is usual now.. It was Paul Bouissac who I met at the first international conference on humeou and laughter in Wales in 176 who told me I was studying text. Since then I have named my papers ‘the Melody of the Text’, Mark 1,2,3…..n The interaction that I recorded in 1976had been unscripted. The ‘immutable synchrony’ of the script as Paul put it was caught by the technology of video tape and hard copy of the movement plots. This was early days for such technology, and it took all night to print with the nose movements of my subjects with the pioneering 3 D print application ‘Picasso’ from the University College London mainframe.

   Maybe Adolfo chose ‘Action words’ for a similar reason to my choice of ‘face’. They are intermediary between the arbitrary signs of the Saussure chess game, and the physical movements themselves, halfway between culture and nature. I am very impressed by this work, which is probing the localization of functions in the motor cortex subserving perception as well as motor output. Adolfo is developing a motor theory for the semantics of action words and found that stimulation of the same pre motor cortex by means of deep electrical stimulation also impaired processing of the ‘action words’.
   Although nerve conduction is much faster, cognitive processes appear to start at 150 mesec and continue past the p300 of ‘cognitive realization’ to the longer melodic shapes of topic and outcome, which is the stuff of interaction. These longer shapes may correlate with short term or working memory, and indeed may be structured. Francis Crick found a role for the Claustrum for bringing it altogether in a present moment, and he thought this necessary because cortical processing appeared to be distributed not localized (5). William James’s realization of the ‘specious’ present pointed out that subjective time comes in chunks. More recent work suggests that the Claustrum may act more like a web browser in relation to information retrieval from distributed locations in the cortex, but the remarkable shared timings of motor movement, perception, must point us to the ‘model that is’ turning out this behavior from the brain.

   I note the dictum from Head which was ‘things that fire together wire together’. I suggested from my study that speech and movement were part of one and the same patterning program, because of the close coordination between the two. How fascinating it was then to hear Alfonso suggest the action words get processed in the motor cortex through a kind of re-run of the motor process involved in the action itself, and faster at 150 msec from fMRI studies than in the anterior temporal cortex where the modalities are bought together. May I also suggest from these timing considerations that there is no time for ‘emotions’ to be added it as a separate process additive to communication of concepts? This together with the emphasis of the cognitive role of the basal ganglia, the thalamus, the amygdala, and the cerebellum have as well in the logical part of verbal communication, suggests that the separation of ‘affect’ from the speech with movement stream may itself be a cultural artefact, and is now an obstruction to achieving the over view that we need. We need a non-dual paradigm for understanding cognitive processing in real life texts.

   By ‘understanding’ I mean modelling, or making sense, of the brain working normally, in context. There are various sorts of sense being made at the same time and Adolfo is trying to mediate between the layers. Sense making is fundamental to our activities as performers in the everyday worlds, and as theorists. I think that ‘making sense’ can be seen as completion or partial completion of models. David Hestenes (2) suggests that Homo Modellens or ‘modelling man’ would be an apt summary of us, and to my mind all questions are incomplete models, and all answers are bits that complete or partially complete a model. It is a model that is shared where there is mutual understanding.

   It is real progress when an association between cognitive process and neurophysiological reality is demonstrated, as it has it been here for action words. However I wonder if Alfonso would agree with the statement attributed to Einstein that is the theory we have which determines what it is we can measure? We need a theory of what the brain is doing, which is especially ironic as we try and make sense of making sense, as Rukmini Nair pointed out. There are a great number of different sorts of sense making activities going on while we engage in spontaneous improvised interaction, and the sense the participants are making to each other is only one. Alfonso admits there are a great many ‘dimensions’, and he usually means ‘variable’, but some of the variables are literally the 2,3, and 4 D co-ordinates that the visual system deals in, which may be ‘hiding’ 5 or higher dimensional processes. The synchrony of execution, verbalisation, and processing suggests that all three are dancing to a tune coming out of a higher dimensional model. The concept of ‘dimension’ and that of ‘variable’ are used interchangeable in modelling, but non-verbal modelling must be especially adapted to the 4D interface which comes from nature, our shared existential predicament. . It is the theory you have which determines what you can measure and what these ‘variables’ are. However at least one study suggests that the brain is modelling in ten dimensions (3) and the limit to subjective understanding to 4 or 5 is an accident of our human condition, and a ‘given’ in the version of the multi-dimensional ‘Brane’ world as described by Stephen Hawking.

   ‘We are constantly predicting what will occur” observed Alfonso and Roman Esqueda. Prediction is by analysis of outcomes in statistical fashion, or from a tightly controlled and thoroughly tested model. Projection is the enactment of an algorithm. Projection is ‘doing it’, and the metaphor is that of an optical projector. With the dimensionality of spacetime being limited to four, additional dimensions in our modelling can project down onto it to control outcomes, and so prediction and projection become the same phenomenon.

   The subject of localization of function in the cortex remains controversial, with the scepticism about the detailed compartmentalization of the cortex from some years back and its replacement with a holonomic theory of non-localised function e.g. Karl Pribram.. Since the work of Hubel and Wisel we have known that the brain de composes vision into numerous features, each with their separate brain areas and recent work has found about 400 anatomically distinguish able zones in the cortex. Work such as Alfonso’s is confirming the importance of it but there is also a distributed nature of cortical representations. How can something be both localised and distributed at the same time? The brain has more mysteries yet.

   An increase in blood flow t the motor cortex at only 150 msec after an ‘action word’ suggests that the motor mechanism is itself the sensor or filter of the incoming stream for embodied motor patterns, which could be called ‘melody’ fragments. It is important to sort out which ‘variables’ are reifications of cultural processes and which are derived from technology. Even instrumental measures from the speech with movement stream contain implicit theories in their design, for example in isolating out the shapes of fundamental frequency and formant analysis which have been extracted algorithmically. I doubt if the AI based language analysis can keep up with the synchronous unfolding of execution, perception and verbal utterance, in terms of timing. It is the exquisite co-ordination of articulation with other supposedly non articular movements such as gesture and head dance in the 4 D space which is remarkable, including the making and breaking of eye contact. It is important to recall that it is stable gaze that is the great achievement of the moving head eye body complex, and these induced head movements may directly invoke mirrored experience in the participants, when they are in eye contact.

   The notion of ‘kinaesthetic melody’ being pre verbal as Roman Esqueda suggests sits well with the ‘motor theory’ of speech production and perception advocated by AM Lieberman, extended here by Adolfo and his colleagues? But we must also keep in mind the natural history of visual behaviour, which is to execute a series of visual ‘takes’ on the environment. The biological organism goes to enormous lengths in terms of compensatory and other specific brain mechanisms such as saccades, following movements, and balance compensation both reflexly and with ‘efference copy’ – all to achieve that momentary stability of the 2 D image on the retina which appears necessary for cognition by quantal sampling. There is a particular brain nucleus implicated as being the still point of this moving world, the nucleus acumbens’, and Crick(4) suggested that the claustrum, a thin sheet of undifferentiated cortex underlying the insula, might have a role in binding disparate events into a single percept, an experience at one point in time. He uses the analogy of the conductor co ordinating a group of players in an orchestra. Could this be the source of the ‘single patterning program’ for both the execution and maybe perception of the complex ‘speech with movement’ that I observed and that Adolfo discusses? Things that fire together wire together…

   In his discussion of introjection, the search for brain manifestations of analytic categories, I think Adolfo was warning against the ‘fallacy of misplaced concreteness’ and suggested doing without them. He mentioned Container/contained and Figure/background,
   to which I add emotion/cognition, feeling/meaning and there are others. It is very hard to ‘unthink’ these, especially the emotion/cognition dichotomy because it appears as self-evident to our world view as the paradigm of an ox cart stuck in the mud was self-evident to Aristotle to underpin his theory of motion (as suggested by Thomas Kuhn.) Adolfo emphasised that the brain miraculously or even ‘viciously’ associates things that happen together. ‘Things that fire together wire together’ appear to include all of the muscular activity involved in the speech with movement stream and now also we learn that the cerebral blood flow changes are synchronised too . What can wire together all these diverse motor organs, and sensations, and cardiovascular changes? Francis Crick has argued that it might be the Clausrum, but that wouldn’t throw them forward in time together. I suggest that it can all work together because it is projective, directly structuring the time ahead to have it ‘turn out’ as the projected outcome and that the modelling going on in discourse is at least five dimension. The virtual dance is visual, evoking and pointing to a shared transitory visualized 4D or more space that is evoked for the participants of ‘speech with movement’, with virtual pointing and glancing at this ephemeral virtual construct

   We are looking for another narrative to look for in the brain which doesn’t assume the dichotomous classification of our experience into picture/frame, emotion/communication. This could liberate us from the prison of ethnocentric concepts and help us get a handle on the most subtle workings of the brain, which are beyond words. I will illustrate with an extension of the example from Rukmini Nair of a logical model switch in: ‘The baby cried and the mother picked it up’ versus: ‘The mother picked up the baby and it cried’.
   Speech melody or prosodic form can be grammatical and disambiguate as well as ‘express emotion’ at the same time. For example: ‘John has plans to LEAVE’ vs ‘John has PLANS to leave’ again achieves such a flip between interpretations of a John that leaves plans, and a John who may depart. Again the cognitive shift is abrupt, but is done entirely by prosody in the john leaving plans example.
   ‘S’ shaped fall and fall-rise
   I will argue that such ambiguity and disambiguation achieved by melody depends on two postulated universals, firstly that the ‘S’ shaped fall goes with the ‘point’ of the utterance and it bears in its trajectory the timing of the cognitive flip in the mind of the hearers. I am suggesting that participants are in a projective mirrored dance which is synchronising their movements and motor apparatus and that it is not prediction that holds it all together but projection in the geometric sense of the text being thrown forward rather than retrieved, projective not derivative. The speech and kinesic melodies are quite literally the trajectory of the transition between the stable states of achieved and perceived plans shared in this group brain that is evoked when people talk. Some work described there had actually been able to measure synchronous activity in the brains of participants to interaction, even suggesting that it is ‘time to move beyond the single brain’ hypothesis to suggest a kind of Syncytium as I suggested in 1977. Adolfo suggested that implicature from is ‘out of scope’ for his enquiry, but it might be amenable to hard science probing if supra individual processes can be demonstrated in conjoined brains. As Alfonso suggests, we are working on the invisible, into areas opaque to words. Damasio was after he same thing when he talked of a movement of proto-self.

   In the above example there are three ‘S’ shaped falls, marked here by capitals
   ‘Let’s get AWAY from telling jokes ‘cos I can NEVER remember any and I’m very BAD at them’

   The ‘S’ shaped fall in Fo goes with the delivery of the ‘point’ in these fragments, which is the same as defining the context in which they occur to have them ‘make sense’. This sense making activity has been well worked up as ‘relevance’, both from the seldom quoted American linguist Richard Gunter in his book ‘Sentences in Dialogue’ (8), and in Relevance Theory which has held the middle ground in studies of implicature for many years (9). Derived ultimately from Grice who first for this tradition emphasised the separate nature of what is said and what is meant and relied on shared intentions and good faith between the parties. This was subtly re defined by relevance theorists into a relevance finding module which has to exist ‘in there’ doing an Occam’s razer job on incoming material, ruthlessly seeking the most economical and thus most relevant interpretation. Approaches to the study of NVC coming from ‘culture’ have ended up in speech act theory, or Relevance Theory, both needing brain ‘modules’ to make them work. This must be what Alfonso means by introjection. In my interpretation of prosody, the ‘S’ shaped fall creates rhythm and is a carrier wave for the models being successively shared, and delivers participants to the same point in the argument at the same time, a cognitive synchrony. The battle for control can be seen in the melody, and is intrinsically teleological, or ‘manipulative’ of the group experience and point of view.

   By considering the kenesic and acoustic melody of a section of text as a musical score with tracks for voice and physical movement in 4 dimensions one can point out natural synchronies and entrainment both within and between people, and their environments, and there is a rhythm to the engagement that resonates even with babies still in the womb. I did not have the entire ‘dance’ to study, but intensive study of 4D was enough for me to discover the implications of higher dimensional modelling.

   We look for new narratives to find in the brain and studies such as Adolfo’s are definitely on the road. A theory will be in words, but might still clearly recognise their limitations. I found a shared definition of our subject matter emerging, a dispassionate account of the human condition and a reintegration of culture in nature as suggested by Levi Strauss, and the resolution of the ‘two cultures’ argument. I believe we now have the outlines of ‘that theory of behaviours and realities that it has taken the species so long to achieve’ from the Melody of the Text, Semiotica, 1980 (9) It would be in the nick of time’, because to paraphrase Marx, whereas the philosophers and other theorists have tried to understand our human world, now we need to know. The present climate and resource crises demand an approach based on ‘true theory’, as ‘whatever the cost and whatever the consequences’ becomes the cry of the climate radical. This might be backed up with a scholarly consensus on human behavior, to find a way that works for us all in the next few crucial decades.

   1. Snow’s ‘two cultures and the scientific revolution’ was delivered as the ‘Rede lecture’ in 1959 at Cambridge UK. Leavis characterised Snow’s book as ‘a document for the study of cliché’. He asserted ‘there is no wealth but life’. https://www.theguardian.com/books/2013/aug/16/leavis-snow-two-cultures-bust
   2. http://modeling.asu.edu/R&E/Hestenes-ConceptualModeling2015.pdf
   3. https://www.sciencealert.com/science-discovers-human-brain-works-up-to-11-dimensions
   4. https://www.sciencedirect.com/science/article/pii/S0960982220314391#:~:text=The%20claustrum%20is%20a%20brain,be%20critically%20linked%20to%20consciousness.
   5. https://www.sciencedirect.com/science/article/pii/S0149763418305530#

   6. https://reader.elsevier.com/reader/sd/pii/S1364661322001346?token=573A11DCDF5724CAC3D69D5C83118ACB59F9B665EBF8FE64585979C9512C9CF910AF2795769229F3DE651419DBC9E2ED&originRegion=us-east-1&originCreation=20221118094002
   7. Richard Gunter 1974 Hornbeam Press
   8. Relevance Theory https://www.sciencedirect.com/science/article/abs/pii/S0378216606000397
   9. https://www.degruyter.com/document/doi/10.1515/semi.1980.32.1-2.119/html
   The Melody of the Text M Mair

   Reply