Ideas

Gimme a beat! (Affordance ontology musings, part 2)

The affordance concept is often (mostly?) discussed with reference to visual perception. For example…

“The chair affords sitting, standing, running around.” “The aperture affords walking passability.” “The gap between approaching players affords running through with the ball.”

In each of these examples and others, the discussion of the properties of objects and events in agent-relevant terms (my shorthand definition of affordances) is usually couched in terms of how they are perceived visually. This then becomes a discussion of how force-resisting properties of substance get into the optic array, or how the temporal gaps formed between moving objects are specified visually, and which invariants in the optic flow do agents use to coordinate with those gaps. Don’t get me wrong, all this is great stuff! But I feel that the affordance concept could be developed further by giving comparable airtime to the pick-up of agent-relevant properties of objects and events through non-visual perceptual systems. Such as audition. Perhaps not surprisingly, my thinking is that asking questions about how actions are coordinated with sound more generally, and music specifically, can enrich and challenge existing formulations of the affordance concept.

An example that I like to think about is a beat. A simple rhythmic, acoustically-specified* beat. Think of the sound of someone tapping a repeated, regular rhythm on a table top or leather couch. Easy to recognise, easy to perform, easy to describe. But when it comes to analysing this in terms of the affordance concept, things actually becomes a little trickier.

On the one hand, affordances are supposed things in the world which constitute ‘opportunities for action’ for a given perceiver-actor. In a sense, a beat is an opportunity to coordinate the movements of one’s body in time with. This might be in an event-to-event matching way (e.g. tapping along, nodding one’s head, stepping in time, dancing in a way that movements start and end to coincide with individual beat sounds). It might also be in a looser tempo-matching way (e.g. dancing more freely from the rhythm, but in a way that movement rates loosely fall into ratios with the beat durations).

On the other hand, Gibson’s derivation of the affordance concept is built upon identifying the ecological content of the environment in terms of substances, their surfaces, and the energy media disrupted by those surfaces, particularly light. For example, a surface which affords support does so in virtue of the spatial-extension and pressure-resisting properties of the substance (relative to the shape, size and mass of the organism), and is perceivable in virtue of lawful relations between the substance’s surface and its interaction with the surrounding ambient light. Similar analyse can be (and have successfully been) applied to passable gaps, interceptable moving objects, avoidable collisions or falls, etc.

Our beat is not clearly like any of these visually-perceived affordances. It is independent of any particular type of substance surface interactions, other than that two substances’ surfaces have to collide regularly (even this is not essential – see below). To illustrate, the same beat could be created by tapping on the couch, banging a spoon against a pot, hitting a bass drum with a foot-pedal, etc. We could even alternate between these different interactions and preserve the beat, as long as the regular temporal pattern between collision events is maintained.

This may not seem an issue, as affordances are ‘invariants of invariants’. A passable aperture, for example, could be constructed from wood, metal, stone, etc. What matters across these different structures of substances is the width and height of the aperture in relation to the width and height (and range of motion of these while locomoting) of the passing creature. Yet, the passable/impassable distinction is still built into the solidity of the substances, and the forces they will resist when coming into contact with a solid, moving organism. Many of the affordances discussed in the context of visual (and haptic) perception share this feature (graspable objects, avoidable projectiles, visual cliffs, etc.). Their meaningfulness is based on possible or actual contact between the perceiver and a substance of some kind.

The beat-as-affordance does not share this feature. It can be action-relevant independent of any substance property or the effect of a substance’s surface on the acoustic energy medium. But tapping on a leather couch is a contact event between two solid substances! Drops in liquid can form a beat. But these are still substance interactions! Digitally synthesised sounds with no material analogy can form a beat.

Arguably, a beat is action-meaningful because of its temporal patterning alone. One could argue that some visually-specified dynamic events, such as the time-to-arrival of an approaching object, are action-meaningful in virtue of their temporal properties (i.e. when they will arrive at the point of observation and whether or not we still want to be there when they do). Nevertheless, these affordances are almost always picked out by their agent-relevant spatial–temporal properties. An audible beat as something which supports coordinated action can be picked out by its agent-relevant temporal properties. No real thing has to get from A to B.

There is much more that I could discuss on this topic: the perceptual information to specify a beat; the action capabilities that might make a beat an affordance; the coordinated behaviours that a beat can support; the species-specific (or not) nature of musical beats; issues to do with skill, culture, style; beats as invitations, and their acceptance or refusal as a marker of agency. I hope to pick up on some of these topic in the near future.

For now, I’ll leave you with one of my favourite Beats and invite you to accept what it offers you for good or ill… 😉

 

—–

* Putting to one side the fact that beats may be visually-specified under some conditions, albeit with less behavioural and phenomenological potency

Ideas

Affordance ontology musings, part 1 (surveying the valley)*

Every since James Gibson coined the term ‘affordance‘, the concept has proved both fruitful and frustrating.

Fruitful because the term encapsulates a revolutionary way to think about what it is we perceive: the things in the world in terms of our abilities and capacities to act as embodied agents. Fruitful because it provides an account of how the things we perceive can be meaningful to us, without the Dualist story of mental life needing breathed into otherwise dead sensations for the benefit of some Cartesian Theatre audience. Fruitful because it offers a way to dissolve the subject-object disjunct that is eschewed by ecological psychologists, 4E’s cognitive scientists, American pragmatists, and other like-minded Monists. And fruitful in its enrichment of the conceptual language of numerous applied fields, such as sports, human-computer interaction, music technology, etc.

Frustrating because the term has been commuted from its original definition to mean something like ‘mental representation of available actions’ by some cognitivist psychologists, neuroscientists and design theorists. Frustrating because the concept has proved tricky to fully institute within a coherent scientific programme of investigation. But most of all, frustrating because the community of researchers invested in understanding and studying affordances can’t agree on what the hell they are!

This reflects both a feature and a bug of the original derivation. Here is the much quoted definition(s) given by Gibson in his 1979 book:

“The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, but the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment.” p127

“…. an affordance is neither an objective property nor a subjective property; or it is both if you like. An affordance cuts across the dichotomy of subjective-objective and helps us to understand its inadequacy. It is equally a fact of the environment and a fact of behavior. It is both physical and psychical, yet neither. An affordance points both ways, to the environment and to the observer.” p129

This would have pleased his philosophical forebears, James, Dewey and Merleau-Ponty, in its disrespect for any presumed subject/object, mind/body distinctions, and associated metaphysical bottlenecks. However, it has created problems for those who wish to respect its Monist overtones while actually fleshing the concept out into something more theoretically precise and scientifically investigable. These problems stem from the idea of affordances as both real features of the environment and real features of how organisms and environments relate to each other.

But surely that’s a good thing right? In capturing both the environment-as-relevant-to-an-organism and the organism-environment-as-singular(complex)-system, the affordance concept can dissolve the dualist rift we all hate and deliver us to monist salvation, no? Given Gibson’s formulation, his intention was apparently ‘yes’! (Feature!) However, the trouble started when others tried to identify which part of the organism-environment system the noun ‘affordance’ should actually be attached to. Should the affordance tag be tacked onto the environment (albeit defined in organism-relevant terms), or onto a loop of two-directional arrows pointing back and forth between the environment and the organisms? If the former, affordances are ‘out there’: features of the environment to be used, avoided, coordinated with by organisms with reciprocal features, but yet having some kind of existence independent of those organisms. If the latter, affordances are the relationships between features of the environments and features of the organisms, coming in and out of existence as those features come in and out of contact with each other. These two interpretations are not compatible, and this has led to much confusion, spilled ink and hurt feelings. (Bug!)

Although this might seem like a merely semantic debate, or a concern for people with too much time and ontological neuroses, it actually causes real issues for those who wish to apply and develop the affordance concept in a rigorous way. If affordances are properties of the environment (dispositions interpretation), then they can structure energy arrays and hence our perception of them can be grounded without appeal to Dualistic concepts like mental inference, etc. However, this has been seen to allow a creeping Dualism through the back door, in assuming a pre-given (objective) world distinct from the acting-but-receptive (subjective) organism. The relations interpretation of affordance avoids this by giving an account in which the environment and organism co-constitute (mutually determine) each other, so the creeping Dualism is securely locked out. However, an apparent consequence of this version is that the ephemeral nature of relational affordances makes it hard to explain how perception could be grounded in anything reliable like lawfully structured energy arrays. If perception is ungrounded, this ungrounds action, which ungrounds behaviour, which ungrounds life, etc. This might be appealing to constructivists, (post-)postmodernists, anti-realists (idealists), and so on, but it does nothing for those of us who still see the affordance concept as having potential explanatory value.

Thus, we survey a valley. The dispositional interpretation at one peak offers a grounded science of perception and action, but fails to tell us much (yet) about how the social, cultural, historical contexts of living organism-environment processes develop. The relational interpretation at the opposite peak engages heartily with the dance of behaviour, situation, culture, history and future, but risks failure to achieve much more than compelling storytelling. The attempts (so far) to bridge these peaks are either incomplete structures aching from one side, or mumbling heaps on the valley floor.

Thank you Jimmy Gibson for introducing the affordance concept into our vocabulary. And cheers for leaving us to sort the goddamn thing out, you sod! 😉

 

* Ed Baggs’ blog post ‘What affordances can’t do‘ covers the exact same topic as this post. Ed’s is shorter and better. The reason I’m going to the bother of writing my own ‘midterm essay’ version is because I want to explore how this debate plays out for things I’m personally interested in for future blog posts, so it seems appropriate that I lay the shaky groundwork for those discussions myself.

Ideas

Some lessons from being my own motor learning study participant

I was recently reminded that it is a good idea to be your own experimental participant.

A PhD student I supervise, Michal Toth, is designing a study to investigate how people learn a new motor task. To help with the study design, I recently spent a few hours over a week and half being a full pilot participant. Doing so was an interesting exercise, not just for informing the study design, but also for reflecting on the relationships between the experimenter’s intentions for a motor learning study and a potential participant’s experience. Here is an overview of the thinking behind the study, followed by some reflections and speculations from my own experience.

Background to the Research Study

First of all, some background to the aims of the study. Motor skills generally require coordination between multiple effectors (meaning limb segments and/or manipulated tools/instruments) to achieve goal (task outcome). This is because there is redundancy (or ‘abundancy’) in the ways that effectors can be coordinated to achieve a task action outcome (Nikolai Bernstein‘s famous insight). To study motor coordination along these lines, changes in effectors can be measured as different execution variables (e.g. joint angles), while the proximity of action outcome to its goal can be measured as a goal variable (e.g. target error). Then, the execution and goal variables can be tracked in relation to each other, by representing changes in the goal variables against corresponding changes in the execution variables. Ideally, a mapping function between execution variables and goal variables can be analytically determined, but even if not, the patterns of co-variance among execution variables that entail successful task actions (minimise goal variable error) can be informative about motor coordination, and its change with practice/training.* These ideas are brought together in the ‘perceptual-motor workspace‘ concept (Newell et al., 1991), which captures the idea of a lawful relationship between the subspace of motor coordination possibilities and their perceivable consequences that characterise a given task.

From a skill learning perspective, this approach to understanding motor coordination throws up some interesting questions:

  • Are there optimal parts of the perceptual-motor workspace that more advanced performers tend towards (and why)?
  • What sorts of trajectories do learners take through the perceptual-motor workspace, and how do their starting points affect this process (individual differences in skill learning)?
  • What perceptual information drives these processes (in both performance and learning)?
  • How might augmented perceptual information (cueing or feedback) influence one’s trajectory through the perceptual-motor workspace, and can this be tailored to an individual’s current history in that space?

Although Michal and I had already been discussing these questions, after reading Matheus Pacheco and colleagues’ excellent discussion and synthesis in their 2019 paper, we were sold that we should pursue this approach in his PhD studying augmented feedback to enhance skill acquisition. Specifically, we want to capture different learning trajectories through the perceptual-motor workspace of a task of our own contriving, such that we can then test out different approaches to augmented feedback design for enhancing novice learning of that same task.

To do this, we needed an experimental task where: a) we could define a redundant relationship between execution variables and goal variable; b) the execution variables were sufficiently few that we could clearly map the goal variable onto them; c) the task required practice to learn. We would then study people learning the task over a longish period to discover the range of novice to quasi-expert pathways that we could use to inform acceleration of skill learning through augmented perceptual information, testing some theoretical hypotheses along the way.

We came up with two tasks, each of which involved controlling a visual computer avatar by coordinating two manual variables (latter hand and thumb pressure on a joystick). My job was to practice each of the tasks for a number of hours each, as a pilot test, to help inform us about which to use in the study. Along the way I had some thoughts which I thought worth recounting.

Attention drifts and so does performance

In spite of being motivated to do well and provide useful data, I found my attention drifting after a few dozen attempts at each task. Weirdly, in one of the tasks (but not the other), the same Belle and Sebastian song kept playing in my head every time! When I noticed this, it became incredibly distracting and would cause me to perform the task poorly due to lack of concentration. Other times, I just zoned out by the 40th-odd attempt and messed up. Although drops in task performance due to attention are an obvious issue, many motor learning tasks are pretty dull, and I don’t know that we necessary worry about this sufficiently in how our experiments are designed and run.

Small mistakes can disproportionately distort learning trends

Related to the above observation, when I made a major blunder during a series of otherwise-good attempts, I made a note of what trial it was to check in the data afterwards. From this, I noticed two things: 1) a seemingly innocuous fluff on the part of the learner can show up as a massively higher error compared to the surrounding trials, creating the appearance of a major drop in performance; 2) it can take a few trials for performance to get back to where it had just been after a mistake, inflating the average error for that block of trials. Thus, for the learner this can seem like a minor stumble in an otherwise good run, yet to the metrics, it can seem like a major deterioration in performance or slowing of learning.

Simple tasks become boring quickly, absent a broader context of implementation

One of the two tasks turned out to be noticeably simpler than the other, and I reached an apparent ceiling (improvements-with-practice reduce to near zero) fairly early on. At this point, there is little task-intrinsic motivation to keep trying to improve the score with increasingly small gains. In the context of skill acquisition, it felt to me that this would be the point that I should start using this newly-acquired task-action in a more skill-relevant context, e.g. by adapting it to different constraints or parameters. I would liken the experience to practicing returning a tennis ball from a machine on a single setting, or switching between barre chords on a guitar. While these techniques can be useful, their value is in being able to pull them off successfully while playing a match or a song (respectively). Without that broader context of implementation to aim for, there’s not much motivation to keep rehearsing them for their own sake.

Explicit knowledge of the perceptual-motor workspace helps a little, but not a lot

Because I was party to the aims of the study, the implementation of the apparatus and displays, and knew how the measures of performance are calculated, I was not exactly a naive participant. Although this kind of ‘declarative’ or ‘descriptive’ knowledge (“knowing that”) is often distinguished from ‘procedural’ knowledge (“knowing how”), I felt it did help my slightly in getting to grips with the tasks. For example, I spotted that something I was doing early in each attempt was impacting my error score substantially, as a result of knowledge about how that score was calculated. I also was able to strategise about how to approach the task on the basis of knowing what was going on in terms of the execution to goal variable mappings. That said, this knowledge was way more effective in earlier stages of practice (consistent with Fitts & Posner’s classic Skill Acquisition model). As practice went on, it felt more like I was controlling the behaviour and outcome of the visual displays through my shaky-but-coupled actions, rather than anything I could verbalise, even with all my knowledge of how the task worked. This is consistent with both classical and ecological/dynamical models of skill acquisition, but was cool to experience first-hand.

The feeling of performing a task after practice is perceptual

One last thought – as I was getting better at both tasks, I realised that my sense of doing the task-actions well, or my sense of them going wrong and needing recovered, was in how I perceived the flow of the situation unfolding and my intention to drive it in a certain way (i.e. achieve the task outcome). In other words, after a while, there was very little awareness of the separate movements or their combinations, but rather the system I had become part of and was trying to influence. I know this is obvious from an ecological and/or enactive way of thinking, but it was worthwhile to undergo that experience from scratch again, rather than just read and write about it. It also reinforced for me the essential role of learning and development in those schools of thought, something I hope to really interrogate scientifically in the next few years.

It’s good to be your own experiment participant!!

 

  • Newell, K.M., McDonald, P.V., & Kugler, P.N. (1991). The Perceptual-Motor Workspace and the Acquisition of Skill. In J. Requin and G. E. Stelmach (eds.), Tutorials in Motor Neuroscience, pp. 95-108.

 

*For a review of some of the methods and analyses used for this kind of research, Dagmar Sternad’s paper on noise and variability in human movement contains a good overview

Ideas

What can we do with sounds? (Part 1)

A few months ago, Andrew Wilson tweeted a hypothesis:

He had previously suggested something similar to me at EWEP15, although not in as strong a form. In this form, the hypothesis is plainly false, given the ambiguity around the word ‘specialise’ and the fact that many animals – including humans – perceive events and properties of the environment through sound, and not always through echolocation (even among bats, not all species use echolocation, or use it as the sole means to localise targets and navigate their environments). However, after replying to say as much, Andrew refined the hypothesis to something that I think is more conceptually interesting and challenging to those of us interested in auditory perception for action:

This question has been rattling around my head since the summer, and I have been meaning to write up my thoughts on here for a while. However, I’ve since realised that the question itself raises a number of different issues and ideas about auditory perception, its functions, and its comparability to other senses. These are relevant to the development of Sensory-Substitution Devices (SSDs), but I think reach into more general questions about auditory perception for action. As such, I’m going to try and tackle this over a number of posts (at some point over the coming year, I promise).

What can we do with sounds and what can’t we do?

To begin with, I thought it might be useful to group some activities, abilities and skills into those that humans specialise using sound to achieve, those that are typically guided by vision but which may be guided/influenced by sound, and those for which sound is pretty much useless compared with vision.

Things that humans do well (specialise) with sound over vision

  • Perceive and control speech (however: McGurk effect)
  • Perceive and make music (however: vision can influence perception of musical performances: 1, 2, 3)
  • Localise events and objects outside visual field, albeit less precisely than through vision

Things that humans do well (specialise) with vision, though may be able to do to a lesser ability using (naturally occurring/non-artificial) sounds

Things that humans do well (specialise) with vision, that are impossible using (naturally occurring/non-artificial) sounds

  • Read text
  • Perceive signs/symbols
  • Anything involving colour perception
  • Recognise faces

This catalogue is far from exhaustive (although I intend to add to it over time), but hopefully it will fuel thinking about what auditory perception may allow us to do and why. In subsequent posts, I intend to consider cases when human uses of sound may vary (as can be the case for people with visual-impairments, or in learning an auditory-based skill like a musical instrument), find out more about acoustic perception and action in non-human species, and explore theoretical ideas which may make sense of these different groupings from the perspective of an ecological approach to acoustic psychology. While I have thought about these questions of-and-on for some time now, this is very much a learning exercise and I do not yet know where my conclusions will end up. Good fun!

Conferences/Workshops

EWEP15: Thoughts and Reflections

Last week I attended (most of) the 15th European Workshop on Ecological Psychology (EWEP) in Mountauban, France. This was my sixth EWEP, having first attended in Madeira in 2008, and for me it is as much of a get together with friends and colleagues from around the world as it is a scholarly meeting. Still, the scientific and theoretical content of the workshop was generally of a high quality, showcasing some interesting empirical and conceptual developments in the field of ecological psychology and related domains.

Some of my highlights of the workshop:

  • As with other recent meetings, there is a strong cohort of people developing the theoretical framework(s) of ecological psychology, both in terms of tackling conceptual controversies as well as in attempting to extend the range of activities and situations that ecological psychologists might meaningfully turn their attention to. Rob Withagen presented ideas, inspired in part by the work of Tim Ingold, about how creativity emerges from doing in the context of art and architecture. Ludger van Dijk drew links between Gibson’s ideas and those of the American Pragmatists. Julian Kiverstein attempted to tackle the problem of what Clark and Toribio (1994) called ‘Representation-Hungry‘ behaviours, activities which seem to necessitate mental representations to fill-in for the absence of behaviourally-relevant environmental properties. Matthieu de Wit presented updates on his project of a ‘Gibsonian Neuroscience’, in which the concept of ‘neural re-use’ supports the re-framing of neural processes as being organised around tasks rather than anatomical regions. Finally, Ed Baggs presented recent work on the distinction between the ecological environment as a habitat for a species, and as an Umwelt for an individual agent. While there are many challenges in advancing ecological psychology along these lines, some of which I hope to discuss in the blog soon, it is great to see people continue to enthusiastically engage with core concepts in these ways and I look forward to tracking their developments.
  • Another pattern I noticed was that a number of presentations featured data on how different individuals performed in different tasks, either as individual cases or in figures which showed the full range of performance/perception of an experimental cohort. Information about individual differences is key to better understanding issues in development, skill acquisition, expertise, disorders, and even just for understanding skilful adaptability in simple perceptual-motor tasks. I am encouraged to explore individual variation more in my own research, so it was inspiring to see how others engaged with the task.
  • On a related note, there were some great pieces on development (e.g. Laura Golenia‘s work on Developmental Coordination Disorder), skill learning (e.g. Daniel Leach‘s work on bimanual coordination and Agnes Henson‘s research on training new speech gestures with augmented visual feedback), and creativity (e.g. Dominic Orth‘s studies of how constraints in training may force learners to search for alternative affordances). I am delighted to see people adopting the ecological approach to engage with these psychological topics, as I think they are key to the future of the discipline.
  • An entertaining talk on the wisdom/stupidity of crowds in relation to perceived Social Identity information by Daniel Richardson, and an excellent keynote on self-organisation of collective motion in ants and humans by Vincent Fourcassié. Both of these presentations served up food for thought on how ecological psychologists might study social coordination beyond the usual dyadic interpersonal movements or sports team behaviour.
  • In general, the discussion of perception is still largely focussed on vision. However, by comparison to recent meetings, there was a greater representation of work involving auditory, haptic, and multi-sensory perception. Since I often find myself having to broaden discussions with ecological psych people away from the ‘visual fixation’ (pun-intended), this was encouraging.

Some room for change for the future:

  • The structure of the meeting was largely oral presentation-based, with these beginning at 8:30am, and finishing around or after 7pm. This was too much! Plus, with 15-minute presentations, it was a real challenge to keep up mentally with the switches between topics. The inevitable run-over of presentations meant that lunch was reduced in duration and a scheduled coffee break between sessions had to be removed. I saw hardly any of the posters due to the poster sessions coinciding with lunch, and the time for discussion with colleagues was shifted mainly to the nights. In my experience, the value of a conference is largely in the discussion, debate and planning with fellow researchers. Therefore, it was a shame that the format gobbled up a lot of the time and mental energy available to do this. This issue is not unique to EWEP, but in chatting to a number of people there emerged a feeling that perhaps there are other ways to communicate our research and make connections that are not so unidirectional and time-consuming. Given the collegiate nature of the broader EWEP community, and our aim to sustain the ecological psychology program, I think it will be important to explore different ways to facilitate discussion and collaboration in future workshops.