|Author||Randall C. O'Reilly|
|Publication||(O'Reilly, Munakata, Frank, Hazy, & Contributors, 2012)|
|First Published||Aug 15 2016|
|Tags||Attention, Spatial Attention, Posner Task, Neglect, Balint's Syndrome, Object Attention|
|Description||This simulation illustrates how object recognition (ventral, what) and spatial (dorsal, where) pathways interact to produce spatial attention effects, and accurately capture the effects of brain damage to the spatial pathway.|
|Updated||15 August 2016, 14 January 2017|
|Emergent Versions||8.0.0, 8.0.4|
- 1 Introduction
- 2 Perceiving Multiple Objects
- 3 The Posner Spatial Cuing Task
- 4 Balint's Syndrome
- 5 Temporal Dynamics and Inhibition of Return
- 6 Object-Based Attentional Effects
- 7 References
This simulation illustrates how object recognition (ventral, what) and spatial (dorsal, where) pathways interact to produce spatial attention effects, and accurately capture the effects of brain damage to the spatial pathway.
Let's step through the structure and connectivity, which was completely pre-specified (i.e., the network was not trained, and no learning takes place, because it was easier to hand-construct this simple architecture). As you can see, the network has mutually interconnected Spatial and Object pathways feeding off of a V1-like layer that contains a spatially mapped feature array. In this simple case, we're assuming that each "object" is represented by a single distinct feature in this array, and also that space is organized along a single dimension. Thus, the first row of units represents the first object's feature (which serves as the cue stimulus) in each of 7 locations, and the second row represents the second object's feature (which serves as the target) in these same 7 locations.
| ⇒ Now select r.wt in the network view and click on the object and spatial units to see how they function via their connectivity patterns. |
The object processing pathway has a sequence of 3 increasingly spatially invariant layers of representations, with each unit collapsing over 3 adjacent spatial locations of the object-defining feature in the layer below. Note that the highest, fully spatially invariant level of the object pathway plays the role of the output layer, and is used for measuring the reaction time to detect objects. This happens by stopping settling whenever the target output (object 2) gets above an activity of .6 (if this doesn't happen, settling stops after 200 cycles).
The spatial processing pathway has a sequence of two layers of spatial representations, differing in the level of spatial resolution. As in the object pathway, each unit in the spatial pathway represents 3 adjacent spatial locations, but unlike the object pathway, these units are not sensitive to particular features. Two units per location provide distributed representations in both layers of the spatial pathway. This redundancy will be useful for demonstrating the effects of partial damage to this pathway.
Perceiving Multiple Objects
Although much of the detailed behavioral data we will explore with the model concerns the Posner spatial cueing task, we think the more basic functional motivation for visual attention is to facilitate object recognition when multiple objects are presented simultaneously. Therefore, we will start with a quick exploration of the network's object recognition capacities as a function of the spatial distribution of two objects. This will provide an introduction to the kinds of interactions between spatial and object processing that can happen using this relatively simple model. Let's begin by viewing the events that we will present to the network.
| ⇒ Click on the tab to view the events -- is in the upper left hand side. |
You should see 3 events. The first event has two different objects (features) present in different spatial locations. Note that the target object has slightly higher activation (i.e., it is more salient), which will result in the reliable selection of this object over the other. The next event has two of the same objects (targets) presented in different locations. Finally, the last event has the two different objects in the same spatial location. As the figure makes clear, recognizing objects when they overlap in the same location is considerably more difficult than when they appear in different locations. Although it is clearly easier to recognize objects if only copies of the same object are present as opposed to different objects, this difference is not likely to be significant for small numbers of presented objects.
Now, let's test these predictions in the model.
| ⇒ Switch back to viewing act in the network window. Do Batch: and in the . |
This will present the first event to the , which will stop settling (i.e., updating the network's activations a cycle at a time) when the target unit's activation exceeds the threshold of .6 in the Output layer.
| ⇒ Then through the remaining events. |
You should have seen that the network settled relatively quickly for the first two events, but was slowed on the third event where the objects overlap in the same region of space (occasionally not so slow on the last one). The graph view next to the network shows the pattern more precisely, showing the settling times (number of cycles to reach threshold) for each event.
| ⇒ You can go back through the events again using to see the dynamics unfolding cycle-by-cycle -- you might want to click the 5 or 10 button to go 5 or 10 cycles per click. |
You should see that overall the network has more difficulty with the objects appearing in the same spatial location, where spatial attention cannot help focus on one object. The overall average cycles by condition are reported in the upper-right table (BatchAvgOutputData) in the network view display (last column contains the cycles).
|6.6: Report the averages for each condition from the table in the network view panel.|
You should have observed that spatial representations can facilitate the processing of objects by allocating attention to one object over another. The key contrast condition is when both objects lie in the same location, so that spatial attention can no longer separate them out, leaving the object pathway to try to process both objects simultaneously.
The Posner Spatial Cuing Task
Now, let's see how this model does on the Posner spatial cuing task, diagrammed in Figure AS.1.
| ⇒ Set input_data in the to StdPosner. Click on the tab to see the input patterns (upper right grid view in that window). |
There are three groups of events shown here, which correspond to a Neutral cue (no cue), a Valid cue, and an Invalid cue. There is just one event for the neutral case, which is the presentation of the target object in the left location. For the valid case, the first event is the cue presented on the left, followed by a target event with the target also on the left. The invalid case has the same cue event, but the target shows up on the right (opposite) side of space. The network's activations are not reset between the cue and target events within a group, but they are reset at the end of a group (after the target). Thus, residual activation from the cuing event can persist and affect processing for the target event, but the activation is cleared between different trial types.
|6.7: How does the influence of the spatial cue affect subsequent processing of the target, in terms of the settling times on each condition? Report average data per condition/group from the table in the network view panel.|
Typical reaction times for young adults (i.e., college students) on this task are roughly: neutral, 370 ms; valid 350 ms; invalid 390 ms, showing about 20 ms on either side of the neutral condition for the effects of attentional focus. These data should agree in general with the pattern of results you obtained (the invalid effect is a bit higher), but to fit the data more closely you would have to add a constant offset of roughly 310 ms to the number of cycles of settling for each trial type. This constant offset can be thought of as the time needed to perform all the other aspects of the task that are not included in the simulation (e.g., generating a response). Note also that one cycle of settling in the network corresponds with one millisecond of processing in humans. This relationship is not automatic -- we adjusted the time constant for activation updating (dt.vm_tau = 7 instead of the default of 3.3) so that the two were in agreement in this particular model.
Effects of Spatial Pathway Strength
Now let's explore the effects of the wt_scale parameters in the on the network's performance, which helps to illuminate how the network captures normal performance in this task. First, let's try reducing Spat_Obj wt_scale.rel from 2 to 1, which reduces the influence of the two spatial pathway layers on the corresponding object pathway layers.
| ⇒ Do a with Spat_Obj set to 1. |
You should find that there is less of an effect for both the valid and invalid conditions. This is just what one would expect -- because there is less of a spatial attention effect on the object system, the invalid cue does not slow it down as much, and similarly it has less effect for the valid case.
| ⇒ Set Spat_Obj wt_scale.rel back to 2, and reduce the influence from the V1 layer to the spatial representations by reducing the value of V1_Spat wt_scale.rel from 0.6 to 0.55, then to 0.5, doing a Batch for each and noting the cycle averages. |
Here the most interesting effect is in the invalid trials, where the network takes an increasing number of cycles as V1_Spat values drop, ultimately not being able to settle on the right answer with in the 200 max cycle limit. As V1 has less effect on the spatial pathway, it becomes more and more difficult for input in a novel location (e.g., the target presented on the opposite side of the cue) to overcome the residual spatial activation in the cued location. This shows that the spatial pathway needs to have a balance between sensitivity to bottom-up inputs and ability to retain a focus of spatial attention over time. This network allows this balance to be set separately from that of the influence of the spatial pathway on the object pathway (controlled by the Spat_Obj parameter), which is not the case with the Cohen, Romero, Farah, & Servan-Schreiber (1994) model.
| ⇒ Set V1_Spat wt_scale.rel back to 0.6 before continuing. |
Close Posner and Retinal Eccentricity
One additional manipulation we can make is to the eccentricity (visual distance) of the cue and target. If they are presented closer together, then one might expect to get less of an attentional effect, or even a facilitation if the nearby location was partially activated by the cue.
| ⇒ In the set input_data to ClosePosner, and , , and then through this case. |
You will see that an overlapping set of spatial representations is activated, and the Batch Run reveals that there is no longer a very reliable slowing for the invalid case relative to the neutral case.
| ⇒ In the switch the input_data back to StdPosner before continuing. |
Effects of Spatial Pathway Lesions
As we mentioned earlier, Posner, Walker, Friedrich, & Rafal (1984) showed that patients who had suffered lesions in one hemisphere of the parietal cortex exhibit differentially impaired performance on the invalid trials of the Posner spatial cuing task. Specifically, they are slower when the cue is presented on the side of space processed by the intact hemisphere (i.e., ipsilateral to the lesion), and the target is then processed by the lesioned hemisphere. The patients showed a 120 ms difference between invalid and valid cases, with a validly cued RT of roughly 640 ms, and an invalidly cued RT of 760 ms. These data can be compared to matched (elderly) control subjects who showed a roughly 60 ms invalid-valid difference, with a validly cued RT of 540 ms and an invalidly cued RT 600 ms.
You should notice that, as one might expect, older people are slower than the young adult normals, and older people with some kind of brain damage are still slower yet, due to generalized effects of the damage. In this case, we are interested in the specific involvement of the parietal lobes in these attentional phenomena, and so we have to be careful to dissociate the specific from the generalized effects. To determine if the patients have a specific attentional problem, we must first find a way of normalizing the data so that we can make useful comparisons among these different groups (including the model). We normalize by dividing the elderly control's data by a constant factor to get the same basic numbers reported for the adult normals (or the model). If there is a specific effect of the brain damage, we should find that the pattern of reaction times is different from the adult normals even when it is appropriately normalized.
To find the appropriate scaling factors, we use the ratio between the valid RT's for the different groups. Ideally, we would want to use the neutral case, which should be a good measure of the overall slowing, but only the valid and invalid trial data are available for the patients and elderly controls. So, to compare the elderly controls with the adult normals, we take the adult valid RT's of 350 ms, and divide that by the elderly control subjects valid RT's of 540 ms, giving a ratio of .65. Now, we multiply the elderly controls invalid RT's (600 ms) by this factor, and we should get something close to the adult normals invalid RT's (390 ms). Indeed, the fit is perfect -- 600 * .65 = 390. The elderly controls thus appear to behave just like the adult normals, but with a constant slowing factor.
However, when we apply this normalizing procedure to the patient's data, the results do not fit well. Thus, we again divide 350 ms by the the 640 ms valid RT's, giving a ratio of .55. Then, we do 760 * .55 = 418, which is substantially slower than the 390 ms invalid times for the adult normals. This makes it clear that the patients are specifically slower in the invalid trials even when their overall slowing has been taken into account by the normalizing procedure. This differential slowing is what led to the hypothesis that these patients have difficulty disengaging attention.
Now, we lesion the model, and see if it simulates the patient's data. However, because the model will not suffer the generalized effects of brain damage (which are probably caused by swelling and other such factors), and because it will not "age," we expect it to behave just like a adult subject that has only the specific effect of the lesion. Thus, we compare the model's performance to the normalized patient values. Although we can add in the 310 ms constant to the models' settling time to get a comparable RT measure, it is somewhat easier to just compare the difference between the invalid and valid cases, which subtracts away any constant factors.
|6.8: Report the resulting averages.|
|6.9: Compute the invalid-valid difference, and compare it with the patient's data, and with the intact network.|
You should have found that you can simulate the apparent disengage deficit without having a specific "disengager" mechanism.
One additional source of support for this model comes from the pattern of patient data for the opposite configuration of the cuing task, where the cue is presented in the lesioned side of space, and the invalid target is thus presented in the intact side. Interestingly, data from Posner, Walker, Friedrich, & Rafal (1984) clearly show that there is a very reduced invalid-valid reaction time difference for this condition in the patients. Thus, it appears that it is easier for the patients to switch attention to the intact side of space, and therefore less of an invalid cost, relative to the normal control data. Furthermore, there appears to be less of a valid cuing effect for the patients when the cue and target are presented on the damaged side as compared to the intact side. Let's see what the model has to say about this.
| ⇒ In the set input_data to ReversePosner, and do , , for this case. |
You should see that the network shows a reduced difference between the valid and invalid trials compared to the intact network. Thus, the cue has less of an effect -- less facilitation on valid trials and less interference on invalid ones. This is exactly the pattern seen in the Posner, Walker, Friedrich, & Rafal (1984) data. In the model, it occurs simply because the stronger intact side of space where the target is presented has less difficulty competing with the damaged side of space where the cue was presented. In contrast, the disengage theory would predict that the lesioned network on the reverse Posner task should perform like the intact network on the standard Posner task. Under these conditions, any required disengaging abilities should be intact (either because the network has not been lesioned, or because the cue is presented on the side of space that the lesioned network should be able to disengage from).
As mentioned previously, additional lesion data comes from Balint's syndrome patients, who suffered from bilateral parietal lesions. The most striking feature of these patients is that they have simultanagnosia -- the inability to recognize multiple objects presented simultaneously (see (Farah, 1990) for a review). Interestingly, when such subjects were tested on the Posner task (Coslett & Saffran, 1991), they exhibited a decreased level of attentional effects (i.e., a smaller invalid-valid difference). As emphasized by Cohen, Romero, Farah, & Servan-Schreiber (1994), these data provide an important argument against the disengage explanation of parietal function offered by Posner and colleagues, which would instead predict bilateral slowing for invalid trials (i.e., difficulty disengaging). The observed pattern of data falls naturally out of the model we have been exploring.
| ⇒ To simulate this condition, first int the set input_data back to StdPosner, and then set lesion_locations to FULL instead of HALF (keep lesion_n_units at HALF), and do . , , and the bilaterally lesioned network. |
|6.10: Report the results of the Batch Run for the bilaterally lesioned network.|
Finally, we can explore the effects of a more severe lesion to the parietal spatial representations, which might provide a better model of the syndrome known as hemispatial neglect (typically referred to as just neglect). As described previously, neglect results from unilateral lesions of the parietal cortex (usually in the right hemisphere), which cause patients to generally neglect the lesioned side of space. We simulate neglect by doing a similar lesion to the unilateral one we did before, but by doing FULL for lesion_n_units to lesion both of the units in each location.
| ⇒ Specifically, In the set lesion_locations back to HALF, and set lesion_n_units FULL, and do . Set input_data to the MultiObj case. Do , , and in the overall . |
Observe that even when the more salient object is in the lesioned side of space, the network still focuses attention on the intact side. Thus, it is specifically neglecting this lesioned side. In the first case, this causes the network to activate the cue object representation, which does not stop the network settling, resulting in a full 200 cycles of settling.
| ⇒ Now set input_data to StdPosner, and then do , , and . |
You will see a similar neglect phenomenon, which makes the network completely incapable of switching attention into the damaged side of space to detect the target (again resulting in the full 200 cycles of settling for the invalid case).
Interestingly, if one does the ReversePosner case, then all attentional effects are completely eliminated, so that the settling times are relatively similar in all three conditions. This is not because the network is incapable of processing stimuli in the damaged side of space -- by looking at the activations in the network you can see that it does process the cue. Instead, the target presented to the good side of space has no difficulty competing with the weak residual representation of the cue in the damaged side. Competition can explain the general tendency for neglect on the grounds that it is very rare to actually have no other competing stimuli (which can be relatively weak and still win the competition), coming into the intact side of space, so that attention is usually focused on the intact side.
The smaller level of damage that produces the slowed target detection times in the Posner task may be more closely associated with the phenomenon of extinction, in which patients with unilateral parietal lesions show neglect only when there is a relatively strong competing visual stimulus presented to the good side of space (e.g., the cue in the invalid trials of the Posner task). Thus, the model may be able to account for a wide range of different spatial processing deficits associated with parietal damage, depending on both the severity and location of damage.
Temporal Dynamics and Inhibition of Return
Another interesting aspect of the Posner spatial cuing task has to do with the temporal dynamics of the attentional cuing effect. To this point, we have ignored these aspects of the task by assuming that the cue activation persists to the point of target onset. This corresponds to experimental conditions when the target follows the cue after a relatively short delay (e.g., around 100 ms). However, the Posner task has also been run with longer delays between the cue and the target (e.g., 500 ms), with some interesting results. Instead of a facilitation effect for the valid trials relative to the invalid ones, the ordering of valid and invalid trials actually reverses at the long delays (Maylor, 1985). This phenomenon has been labeled inhibition of return, to denote the idea that there is something that inhibits the system from returning attention to the cued location after a sufficient delay.
Our model can be used to simulate at least the qualitative patterns of behavior on the Posner task over different delays. This is done by varying the length of cue presentation (a variable delay event could have been inserted, but residual activation would persist anyway, and varying the cue length is simpler), and turning on the adaptation current, which causes neurons that have been active for a while to "fatigue" (see Temporal Dynamics for details). Thus, if the cue activation persists for long enough, those spatial representations will become fatigued, and if attention is subsequently directed there, the network will actually be slower to respond. Also, because the spatial activations have fatigued, they no longer compete with the activation of the other location for the invalidly cued trials, eliminating the slowing.
Now, let's see this in the model.
You should observe the now-familiar pattern of a valid facilitation and an invalid slowing (although quite a bit weaker due to smaller number of cue cycles).
| ⇒ Batch with increasing durations (change using the cue_dur field) in increments of 50 from 50 to 200. |
You should see that the valid-invalid difference decreases with increasing duration, equalizes at 150 epochs, and by 200 epochs, the validly cued condition is actually a tiny bit slower than the invalidly cued one, which is the hallmark of the inhibition of return phenomenon (Figure 8.28). The effect sizes here are fairly small because the form of adaptation here is relatively weak -- a more significant GABA-B like delayed inhibition effect (which is not currently implemented) would be needed to produce more substantial effects.
| ⇒ cue_dur at 200. through the running of the network with |
|6.11: Report in detail what happens on the valid and invalid trials that produces the inhibition of return effect.|
Object-Based Attentional Effects
So far, we have explored spatially mediated attentional effects. However, the very same mechanisms (and model) can be used to understand object-based attentional effects. For example, instead of cuing one region of space, we can cue one object, and then present a display containing the cue object and another different object, and determine which of the two objects is processed more readily. By analogy with the Posner spatial cuing paradigm, we would expect that the cued object would be processed more readily than the non-cued one. Of course, one would have to use different, but similar cue and target objects to rule out a target detection response based on the cue itself.
Because a simple object recognition task is problematic, the object-based attention studies that have been run experimentally typically involve a comparison between two operations (e.g., detecting a small visual target) on one object versus two operations on two different objects. If the spatial distances associated with these operations are the same in the two conditions, any difference in reaction time would indicate a cost for switching attention between the two objects. Such an object cost has been found in a number of studies (Duncan, 1984; Vecera & Farah, 1994; Mozer, Zemel, Behrmann, & Williams, 1992).
In the simulator, we can run the simpler cuing experiment analogous to the Posner task because we have transparent access to the internal representations, and can measure the object processing facilitation directly.
| ⇒ In the set input_type to ObjAttn, and click on to see these patterns (lower left grid view). |
The first event is a control condition, where we present two objects without any prior cuing. Note that, as in the MultiObj case, the target object is more strongly activated than the cue object, so the network will process the target object. The next event is a cuing event, where the cue object is presented in the central location. Then, within the same group so that activations persist, the next event presents the two objects just as in the first event. Thus, if the prior object cue is effective, it should be able to overcome the relatively small difference in bottom-up salience between the two objects, so that the network processes the cue object and not the target. Finally, the next two events are for the case where the two objects appear in the same location. Recall that before, the network was unable to select either object for processing in this case, because they are spatially overlapping. Perhaps now, with the object-based attentional cue, the network will be able to focus on the cue object.
| ⇒ Make sure adapt_on is off and cue_dur is back to 200, and then do , (you may also need to turn the network display on). Note how the network responds to each of the three task conditions as you continue to Step through these cases. |
You should observe that the prior object cue is indeed capable of influencing subsequent processing in favor of the same object. Note also that the spatial system responds to this in the appropriate manner -- it activates the spatial location associated with the cued object. Finally, note that the top-down object cue is sufficient to enable the system to select one object (even the less active one) when the two objects are presented overlapping in the same location.
| ⇒ You may now close the project (use the window manager close button on the project window or menu item) and then open a new one, or just quit emergent entirely by doing menu option or clicking the close button on the root window. |
- Cohen, J. D., Romero, R. D., Farah, M. J., & Servan-Schreiber, D. (1994). Mechanisms of Spatial Attention: The Relation of Macrostructure To Microstructure in Parietal Neglect. Journal Of Cognitive Neuroscience, 6(4), 377–387.
- Coslett, H. B., & Saffran, E. (1991). Simultanagnosia. To see but not two see. Brain, 114, 1523-1545.
- Duncan, J. (1984). Selective attention and the organization of visual information. Journal Of Experimental Psychology: General, 113, 501-517.
- Farah, M. J. (1990). Visual Agnosia. Cambridge, MA: MIT Press.
- Maylor, E. (1985). Facilitatory and inhibitory components of orienting in visual space. In M. I. Posner & Marin, O. S. M. (Eds.), Attention and Performance XI. Hillsdale, NJ: Lawrence Erlbaum Associates.
- Mozer, M. C., Zemel, R. S., Behrmann, M., & Williams, C. K. I. (1992). Learning to segment images using dynamic feature binding. Neural Computation, 4, 650–665.
- Posner, M. I., Walker, J. A., Friedrich, F. J., & Rafal, R. D. (1984). Effects of Parietal Lobe Injury on Covert Orienting of Visual Attention. Journal Of Neuroscience, 4, 1863-1874.
- Vecera, S. P., & Farah, M. J. (1994). Does visual attention select objects or locations? Journal Of Experimental Psychology: General, 123, 146-160.