By Michael Mair, November / December 2022
I was glad to hear the talk given on the 29th October by Adolfo Garcia. It was an honour also to be counted among the panellists. I am sorry my comments are so late.
Adolfo’s project is trying to heal the rift between the two cultures of ‘personal understanding’ and ‘scientific understanding’. I have had a similar project, but the tool kit described by Adolfo had mostly not yet been invented when I was working on it in the ‘70s. The ‘two cultures’ as a problem followed from controversy around a book by the author CP Snow: ‘The Two Cultures and the Scientific Revolution’ 1959. His thesis was opposed by the charismatic UK literary critic FR Leavis who took a more holistic view (see note 1. For discussion of this). It crystallised out for me as the problem of actually finding a true connection between personal and scientific understanding, and I was able to combine physiological and anthropological perspectives. The destination was scoped by Levi Strauss in ‘La Pensee Sauvage’, the reintegration of culture in nature in all its physical and chemical conditions.
I had a multi-dimensional approach, using a ‘temporal microscope’ to study a number of parameters from two people chatting. I had commenced with images of the 2 D human face because I thought it would bring up all the methodological problems in studying behaviour, while at the same time having a well-defined physical subject matter (Royal Anthropological News Article ‘What do Faces Mean’ 1975). I quickly found out that face and head movements were going ‘with’ speech, and that I had complexity enough just studying head movement. I recorded half an hour of spontaneous chat between 2 students, naïve to the purpose of the recording, but plied with wine and general benevolence. I had an impedance measure of fundamental frequency, two dimensional nose movement plots allowing a three dimensional trace in 2D using perspective, phonetic analysis aligned precisely with the video record by visual inspection of plosives, and a transcription in natural language. There were eight tracks in all (2 people) and I made an interactional ‘score’ of it. I arranged the modalities with the vertical divisions at 1/10 second, and horizontal tracks for Fo , nose movement, phonetic stream and natural language, and there was the taped record itself, I watched it thousands of times in a method suggested by Condon and Ogston, which Mary Douglas put me on to. I was aware that my 3 D nose movement plots lacked a 4th dimension, ‘z’ but I had ‘x’ ‘y’ and ‘t’ for a 3D effect plot. Adolfo is quite right, it would have been better to have had the whole ‘dance’.
My subjects were trying to make sense to each other, but laughs, silences, and extraneous noises in the immersive context were all part of the melody. The ‘control’ for this experiment was ‘uncontrol’, which felt dangerous in 1976, but would be usual now. I met Paul Bouissac at the first international conference on humour and laughter in Wales in 1976 who told me I was studying ‘text’. I had called it an ‘evolving model’. Since then I have named my papers ‘the Melody of the Text’, Mark 1,2,3…..(n). The ‘immutable synchrony’ of the script as Paul put it was caught by the technology, but the interaction had been unscripted. It took all night to print the nose movements plots of my subjects with the pioneering 3 D print application ‘Picasso’ from the University College London mainframe printer.
Now we are getting some hard neurological evidence from Adolpho’s group on how one part of the lexicon – ‘action words’ – are processed in the brain. Maybe Adolfo chose them for a similar reason to my choice of ‘face’. They are intermediary between the arbitrary signs of the Saussurian chess game, and the physical movements themselves, between culture and nature. My data suggested that speech and movement were part of one and the same patterning program, because of the close coordination between the two. They ‘fired together’. Adolfo is suggesting the action words get processed in the motor cortex through a kind of re-run of the motor process involved in the action itself. This pre motor cortical activity is faster at 150 msec than the response evoked from anterior temporal cortex, where the modalities are brought together. This was proven by the timing of blood flow changes from fMRI studies. Although nerve conduction is measured in milliseconds, cognitive processes appear to start at 150 msec and continue past the 300 msec of ‘cognitive realization’ to the longer melodic shapes of topic and outcome, which is the stuff of interaction.
Longer melodic shapes may correlate with working memory and are often highly structured by cultural imperatives. Eye movements flick along at the rate of about 200 msec per saccade, and the cortical response at 300sec – the p300 – has long been assumed to be cognitive. The 150 msec found by Adolfo for processing of ‘action words’ in the premotor cortex is down in amongst the timing of cognitive processes themselves. The remarkable shared timings of motor movement, cortical blood flow, and perception appear to be local effects in the premotor cortex. There may be no time for ‘emotions’ to be added it as a separate process. This together with the cognitive role of the basal ganglia, the thalamus, the amygdala, and the cerebellum often implicated in ‘emotion’ suggests that the separation of ‘affect’ from ‘information’ the speech with movement stream may itself be a cultural artefact, and an obstruction to achieving the over view that we need.
By ‘understanding’ I mean modelling or making sense. David Hestenes (2) suggests the epithet ‘Homo Modellens’ or ‘modelling man’ as a summary of our enhanced aptitudes.. When a model is shared there is mutual understanding. I want to suggest that all questions are incomplete models, and all answers are bits that complete or partially complete a model. The unit of sense may be thought of as the question and the answer, the question an ‘incomplete model’ and the answer the ‘bit that completes or partially completes’ it. However there are often many sorts of sense being made at the same time and Adolfo is trying to mediate between the layers. Sense making is fundamental to our activities as performers in the everyday worlds, and as theorists too, and involves logic in use. A doctor or therapist or ritual operator is making several different sorts of sense at the same time. It is especially ironic, as Rukmini Nair pointed out when we try and make sense of making sense.
It is real progress when an association between cognitive process and neurophysiological reality is demonstrated, as it has it been here for action words. However I wonder if Alfonso would agree with the statement attributed to Einstein that is the theory we have which determines what it is we can measure? We need a theory of what the brain is doing. Alfonso says there are a great many ‘dimensions’ to the study of making sense. For a vision dominated primate important variables are literally the 2,3, and 4 Dimensional co-ordinates that the visual system deals in. Non-verbal modelling is especially tuned to the 4D interface which comes from nature, and is our shared existential predicament. It is the theory you have which determines what you can measure and what these ‘variables’ are. The synchrony of execution, verbalisation, and processing suggests that all three are dancing to the same tune simultaneously. One study suggests that the brain is modelling in ten dimensions (3) and the limit to subjective understanding to 4 or 5 is an adaptation to 4D space/time and an arbitrary limit to modelling.
I noticed the discussion of ‘prediction’. ‘We are constantly predicting what will occur” observed Alfonso and Roman Esqueda in their discussion. ‘Projection’’ and ‘prediction’ are both involved in controlling the future, but for me the metaphor is that of an optical projector. With the dimensionality of spacetime being limited to four, additional dimensions in our modelling can project down onto 4 D spacetime and help control outcomes. From a purely kinesic point of view, it takes a five dimensional model to intercept a four dimensional movement such as a 4 D trajectory, so 5 D modelling must be common in the natural world. For example a surgeon projects the outcome onto the tissues by direct modelling in 4D,which predicts and projects the outcome.
An increase in blood flow to the motor cortex at only 150 msec after an ‘action word’ suggests that the motor mechanism is itself the sensor or filter of the incoming stream for embodied motor patterns. The subject of localization of function in the cortex remains controversial, with the scepticism from some years back and its replacement with a holonomic theory of non-localised function e.g. Karl Pribram. However since the work of Hubel and Wiesel we have known that the brain de-composes vision into numerous features, each with their separate brain areas and recent work has found about 400 anatomically distinguishable zones in the cortex. Work such as Alfonso’s is confirming the importance of localisation of function but there is also a distributed nature of cortical representations. How can something be both localised and distributed at the same time?
The notion that ‘kinesic melody’ is apt for non-verbal motor patterns works well with the ‘motor theory’ of speech production and perception following from A.M. Lieberman in 1967. A motor theory is apt for vision too. The natural history of visual behaviour is to execute a series of visual ‘takes’ on the environment. The biological organism goes to enormous lengths in terms of compensatory and other specific brain mechanisms such as saccades, following movements, and balance compensation both by feedback loops and with ‘efference copy’, all to achieve that momentary stability of the 2 D image on the retina which appears necessary for cognition by quantal sampling. The experience of stereo 3 D vision for those who have it is an illusion by the brain using trigonometry on the discrepancy between images from the two eyes. The illusion of a stable visual world is said to be put together in the Nucleus Acumbens.
In his discussion of introjection, I think Adolfo was warning of Whitehead’s ‘fallacy of misplaced concreteness’. It is important to sort out which ‘variables’ are reifications of cultural processes and which are derived from technology. Even instrumental measures from the speech with movement stream contain implicit theories in their design, for example in isolating out the shapes of fundamental frequency and formant analysis which have been extracted algorithmically. Adlofo mentioned the dichotomies container/contained and figure/background, to which I add emotion/cognition, feeling/meaning and there are many others expressing the same divide. It is very hard to ‘unthink’ these, especially the emotion/cognition dichotomy because it appears self-evident to our world view as was the paradigm of an ox cart stuck in the mud was self-evident to Aristotle to underpin his theory of motion (as suggested by Thomas Kuhn). It is the theory you have which determines what you can measure and what the ‘variables’ are. We are looking for a better narrative about the brain that could liberate us from the prison of ethnocentric concepts and help us get a handle on its most subtle workings.
Adolfo emphasised that the brain ‘viciously’ associates things that happen together. ‘Things that fire together wire together’, not only the muscular activity involved in the speech with movement stream but the cerebral blood flow changes too . The virtual dance is visual, evoking and pointing to a shared transitory visualized 4D space shared by the participants of ‘speech with movement’. The body head eye complex may be engaged in virtual pointing and glancing at this ephemeral virtual construct, which is mirrored among the participants to discourse.
I would like to discuss a ‘non dual’ interpretation of prosody. Rukmini Nair demonstrated a logical model switch with: ‘The baby cried and the mother picked it up’ versus: ‘The mother picked up the baby and it cried’. This is a lexical switch. But speech melody can by itself act grammatically and disambiguate. For example: ‘John has plans to LEAVE’ vs ‘John has PLANS to leave’ again achieves such a flip between interpretations of a John that leaves plans, and a John who may depart. The cognitive shift is abrupt, like a ‘Necker’ cube flip. I suggest that the ‘S’ shaped fall in speech melodies has an important role in the establishment of ‘states of play’ in the evolving model of the interactive text.
I argue that such ambiguity and disambiguation achieved by melody depends on two postulated universals, firstly that the ‘S’ shaped fall goes with the ‘point’ of the utterance and it bears in its trajectory the timing of the ‘cognitive change point’ in the minds of the hearers. This is context bound and it acts on whatever is there, to deliver the next, and so on. Emphatic excursions may be needed to shift cognitive impasses, but they work the same way, without invoking a duality of form and feeling. Secondly I suggest that participants can be in a projective mirrored dance which is synchronising their movements and motor apparatus . It is not prediction that holds it all together but ‘projection’ in the geometric or optical sense of the text being thrown forward rather than retrieved, projective not derivative. The speech and kinesic melodies are quite literally the trajectory between the stable states of achieved and perceived plans shared during conversation. Some work described in a paper on mirroring (6) has actually been able to measure synchronous activity in the brains of participants to interaction using f MRI, suggesting that it is ‘time to move beyond the single brain’ hypothesis to suggest a kind of ‘syncytium’ of participating brains and bodies. Adolfo said that implicature is ‘out of scope’ for his enquiry, but it might be amenable to hard science probing if supra individual processes can be demonstrated in conjoined brains with fMRI. We are working on invisible areas opaque to words. Damasio was after the same thing when in his discussion of the ‘proto self’.
The ‘S’ shaped fall in fundamental frequency goes with the delivery of the ‘point’, and its interpretation is the context in which it ‘makes sense’. This sense making activity has been well worked up as ‘relevance’, both from the American linguist Richard Gunter in his book ‘Sentences in Dialogue’ (7), and in Relevance Theory from Sperber and Wilson which has held the middle ground in studies of implicature for many years (8). It was Grice who first emphasised the separate nature of what is said and what is meant and he thought that successful communication needed shared intentions and good faith between the parties. Gunter suggested that we actually hear the melodic shapes differently according to the relevance we ascribe to it from contextual features in the dialogue (or multilogue). Relevance theorists assumed there is a relevance finding module which has to exist ‘in there’ doing an Occam’s razer job on incoming material, ruthlessly seeking the most economical and thus most relevant interpretation. This may be what Alfonso means by introjection – looking in the brain for validation of a purely conceptual model. In my interpretation of prosody, the ‘S’ shaped fall creates rhythm and is a carrier wave for the models being successively shared, to which it delivers participants at the same point in the argument at the same time, a cognitive synchrony. The battle for control can be seen in the melody. In this way we can have a single theory for melody which can model both the role of it in grammar and in managing ambiguity as well as the so called ‘expressive’ role. The effect of melodies is immediate, the verbalisation post hoc. We don’t need dualistic words for the brain.
By considering the kinesic and acoustic melody of a section of text as a musical score with tracks for voice and physical movement in four dimensions, natural synchronies and entrainment both within and between people and their environments are demonstrated easily. There is a rhythm to the engagement in human interaction that resonates even with babies still in the womb.
I believe we now have the outlines of ‘that theory of behaviours and realities that it has taken the species so long to achieve’ which I hoped for in 1977 (the Melody of the Text, Semiotica, 1980 Mair) (9), and suggested that ‘time would tell’ if it emerged, and it has! The task of achieving a ‘general theory’ of behaviours and realities is now urgent but I think we have the right paradigm. The philosophers and other theorists have only tried to interpret our human world, and now we need to know. Adolfo’s work is a significant contribution to the global effort to model the brain and behavior, and my comments are a distant echo.
Michael Mair
November/December 2022
Acknowledgements:
I am grateful to Adolfo Garcia for his webinar and Paul Bouissac for involving me and enabling me to make these comments and for our work and friendship in the ‘70s still ongoing. My friend John Kirkland from Massey University New Zealand facilitated my data harvest in the ‘70s and we did some papers. He is not responsible for the views expressed here. My friend Carl Urion from the University of Alberta was developing similar ideas on implicature in the ‘70s and taught me that about allowance of control. I am indebted to the UK Social Science Research Council for financial support in ’76-’77 and Mary Douglas of UC London for recommending my project to them.
Refs:
- Snow’s ‘two cultures and the scientific revolution’ was delivered as the ‘Rede lecture’ in 1959 at Cambridge UK. Leavis characterised Snow’s book as ‘a document for the study of cliché’. He asserted ‘there is no wealth but life’. Guardian article from 2013 https://www.theguardian.com/books/2013/aug/16/leavis-snow-two-cultures-bust
- http://modeling.asu.edu/R&E/Hestenes-ConceptualModeling2015.pdf
- https://www.sciencealert.com/science-discovers-human-brain-works-up-to-11-dimensions
- https://www.sciencedirect.com/science/article/pii/S0960982220314391#:~:text=The%20claustrum%20is%20a%20brain,be%20critically%20linked%20to%20consciousness.
- https://www.sciencedirect.com/science/article/pii/S0149763418305530#
- Mirror neurones 30 years later
- Richard Gunter ‘Sentences in Dialogue’ 1974 Hornbeam Press
- Relevance Theory, Sperber and Wilson https://www.sciencedirect.com/science/article/abs/pii/S0378216606000397
- ‘The Melody of the Text’ M Mair’ review of Paul Bouissac ‘Circus and Culture’ Semiotica 1980
https://www.degruyter.com/document/doi/10.1515/semi.1980.32.1-2.119/html
Be the first to comment