AXTut PfcBg

From emergent
Jump to: navigation, search
← previous CPTAX_Program AX Tutorial


  • PRELIMINARY INSTRUCTIONS: This tutorial is a continuation of CPTAX_Program, which is itself an advanced continuation of the basic AX Tutorial. This tutorial assumes the user has a level of familiarity with editing in Emergent, creating simple projects, and so on, as would be acquired by completing both the basic AX Tutorial, and the extra CPTAX_Program segment. Accordingly, these instructions do not go into the nitty-gritty of finding certain elements and doing basic operations, details covered in the basic tutorial. Users are referred there for basic click-by-click-level instructions.
  • Also, the safest place to start for this tutorial is to open the ax_tutorial_cptax.proj and find this AX_Tut_PfcBg page from there. The ax_tutorial_cptax.proj reflects the state of the project after starting with ax_tutorial.proj and having completed the AX Tutorial correctly and then the CPTAX_Program. Thus, ax_tutorial_cptax.proj is a reliable basis for extending it further here. Alternatively, users continuing directly from the CPTAX_Program can continue working in their local, working version of the project. In this case, it is advisable to save that version of the project with a new name (e.g., ax_tutorial_my_cptax.proj) and then work in their customized version.
  • NOTE: The original ax_tutorial.proj itself, as downloaded from the wiki or opened from the …/demo/leabra folder of Emergent, is essentially an empty project with no specific content; the ax_tutorial_final.proj reflects the completion of the basic tutorial; and, the ax_tutorial_cptax.proj reflects the completed state of CPTAX_Program. ax_tutorial_cptax_pfcbg.proj reflects the completed state of this PfcBg segment and is therefore not suitable for working in, but serves as a benchmark for comparison to your own interim and/or completed version.

Adding a Prefrontal Cortex, Basal Ganglia Working Memory System

There are several ways to give a neural network some amount of working or active memory, to hold on to prior events. Perhaps the simplest is to add a "simple recurrent network" (SRN) context layer that holds on to the prior time step's hidden layer activations, and then feeds back into the hidden layer to provide context for the current inputs.

However, there are limitations of this simple SRN memory, which can be overcome by having an active gating mechanism that determines when to hold onto information and when to forget it. One leading scientific theory is that the basal ganglia provide this function, by interacting with the prefrontal cortex, which is widely regarded as the brain area responsible for holding onto working memory. One specific implementation of this idea, called PBWM (prefrontal-cortex basal-ganglia working memory; O'Reilly & Frank, 2006, Neural Computation) is available through the Leabra wizard, and we'll use that.

First, to prepare the model for the addition of the PBWM components, we need to clean up the Network_1 net view panel we'll be using. Since our network is going to end up being a lot bigger, we'll make some more room by getting rid of the extra elements there. Click on the green border of the EpochOutputData Graph View element - it should turn red to indicated it has been selected. Do a Ctrl+d to delete it. Then do the same for the TrialOutputData Grid View element. Zoom and center the network a little to make it easier to work with. Context-click on the far left tab at the bottom we had previously renamed "MainView" and click Save View to save your changes.

Next, we want to move the existing Input layer up to the same level as the Output layer. For anatomically-inspired reasons, PBWM locates various brain-stem dopamine systems in the lower level of the model. To do this, go the left Navigator tree and select the Output layer near the bottom under the layers group of Network_1. In the purple edit panel that appears find and change the pos_abs x index to 5 (from 0). Hit Apply. Next, find the Input layer right above that and change its pos abs z index to 2 (Apply). We have just freed up the whole lower level (z = 0) for the PBWM components.

Next, go to the LeabraWizard_0 (top of Navigator tree on the left), and scroll down to select the PBWM option under Network/Specialized Networks. After it's clicked a dialog with several options and lots of information comes up. Since we only need one stripe enter 1 in both the pfc_gp_x and pfc_gp_y fields that comes up. All the other defaults are okay so leave everything else as is. This will create a PFC working memory with one maintenance stripe, along with a corresponding one for outputting . When you hit OK, you'll get a series of dialogs with information -- just keep hitting OK until it is done. You will see a much more elaborate network now, with many more layers.

TODO: starting here, a whole new overview/theoretical summary of PBWM...and PVLV

Understanding PBWM From 35,000 Feet

PBWM is a very complex model and we can only provide a very cursory, high-level summary here that hopefully provides the user with the gist of what it is doing. As you may have noticed while the wizard was adding PBWM to our network, it is actually two distinct modules with the PVLV module added first, which is nested within PBWM itself. Here is a very brief functional description to give you at least a basic idea of what you will be seeing when we run the network later.

PFC Layers

Toward the top right aspect of the network you should notice four new "PFC" layers, PFCmnt and PFCout prominently visible and PFCmnt_deep and PFCout_deep lined up behind them. Collectively, these four layers are intended to correspond to two cortical macrocolumns with the two front layers corresponding to the superficial (supragranular) cortical laminae (predominantly laminae 2/3) and the two rear layers their deep (infragranular) counterparts (laminae 5a,5b, and 6). The 2 X 2 relational structure reflects two orthogonal divisons-of-labor the model captures:

  • the superficial/deep (front/back) division-of-labor captures a functional distinction between ungated cortical processing (superficial layer only) and post-gated processing (both superficial and deep).
  • the mnt/out (left/right) division-of-labor captures a functional distinction between input-side processing (which includes robust active maintenance; PFCmnt/PFCmnt_deep) versus output-side processing (PFCout/PFCout_deep) with the former having preferential access to sensory-side afferent (i.e., from "upstream" representations) and the latter to motor-side efferents ("downstream" representations). This functional division-of-labor between input-side and output-side processing is motivated by the extraordinarily elegant and rigorously-detailed mapping of the frontal eye fields in the monkey performed by Sommer & Wurtz (2000) that documents this basic idea.

As motivated by the above 2 X 2 division-of-labor, then, the basic flow of new ("bottom-up") information through the four PFC layers proceeds first into PFCmnt, followed (only if gated) into PFCmnt_deep, which in turn flows to PFCout (in a one-to-one, unit-to-unit, representation-preserving manner), and finally (again, only if gated in a second, separate gating event) to the PFCout_deep layer. Critically, only the latter PFCout_deep layer (among PFC layers) has efferent (downstream) access to the actual Output layer (via Hidden) so as to influence the networks actual response.

GPi and GPeNoGo Layers

Towards the bottom right of the network you will see paired layers GPi and GPeNoGo. GPi stands for "globus pallidus, internal segment" and represents both the GPi and the "substantia nigra, pars reticulata" of the basal ganglia (BG), which are together considered to be the exclusive output substrates of the BG, also sharing an identical histology as well as ontogenetic origins. PBWM's GPi also represents corresponding thalamic cells in the model, simplifying what is actually an inhibitory disinhibition of the thalamus by the BG into a more straightforward excitation. Thus, activity in any particular GPi unit reflects a gating event for the corresponding PFC macrocolumn (mapped topologically such that the left GPi unit gates the PFCmnt/PFCmnt_deep macrocolumn and the right GPi unit the PFCout/PFCout_deep one.

The GPeNoGo layer represents the "globus pallidus, pars externa" and conveys counteracting do-not-gate signals that we don't really need to worry about here.

MatrixGo and MatrixNoGo Layers

Finally, we have the MatrixGo and MatrixNoGo layers (just to the left of the GPi and GPeNoGo layers) which are where the learning occurs that ultimately determines the GPi gating signals. "Matrix" refers to the matrix compartment of the striatum where Type 1 dopamine receptor-expressing principal cells define the direct or Go pathway that produces gating events and Type 2 dopamine receptor-expressing principal cells define an indirect or NoGo pathway that opposes gating. Afferent connections into the MatrixGo and MatrixNoGo layers learn based on phasic dopamine signals produced by the PVLV module, which is PBWM's reinforcement learning module for computing reward prediction error signals.

PVLV Module

The PVLV module is extremely complicated in its own right and far beyond the scope of this tutorial. It is composed of almost two dozen layers by itself all of which are towards the bottom left of the network as built by the wizard. For the purposes of this tutorial we will highlight only the VTAp layer for now, which is the fourth layer from the left along the bottom row of all the PVLV layers. It stands for "Ventral Tegmental Area (VTA), positive valence" and is PVLV's representation of all of the typically-responding dopamine cells of the VTA and substantia nigra, pars compacta as characterized by Wolfram Schultz and others. These cells respond with phasic burst firing for positively-valenced events ("good stuff"; red activity in the model) and phasic pauses in tonic firing for negatively-valenced events ("bad stuff"; blue activity).

We will now proceed with editing our project to use the PBWM module to solve the CPTAX task.

Setting the RewTarg Input

Before we can run the model, we need to do one key bit of configuration. The PBWM model learns from rewards and punishments generated based on how it is performing the task. We're going to assume that only reward values generated on the probe trials are relevant, so we need to tell the model when the relevant trials are. This is done using a RewTarg layer, which is a new input layer that was added by the wizard. When this unit's activation is set to 1, that tells the network that this is a trial when reward should be computed based on the difference between the network's output and the correct answer. Note that this is not the direct value of the reward itself, just the indicator of when reward is available and therefore should be computed based on performance.

To implement this RewTarg mechanism do the following:

  • First, go to the LeabraWizard_0 and select Data/UpdateInputDataFmNet -- this will automatically reconfigure your StdInputData table to include a RewTarg column along with several others we won't worry about. (TIP: This function adjusts the StdInputData to any other changes you might make in your network -- a very useful function!).

Next, we need to modify our CPTAXGen program to set this RewTarg input value correctly. This can be done simply as follows:

  • Find the SetMatrixFlatVal(1,"Output",-1,O_N) program statement in the block of your CPTAXGen program that sets the cue input. Duplicate (Ctrl+m) this statement and edit the new instance's arguments as follows: SetMatrixFlatVal(0,"RewTarg",-1,0).
  • Now do the same for the probe block below that, with the same arguments except that the first one (Variant& val) should be set to 1.

Finally, we need to make a few changes to the network itself:

  • In the Navigator tree (left panel) find and expand the Output layer (not the spec), and then double-click or otherwise open the projections group under that. Delete (Ctrl+d) the Fm_PFCout_deep projection. TIP: Alternatively, you could also just lesion the projection by clicking the off box (i.e., off = true flag) in the goldenrod-colored Fm_PFCout_deep control panel.
  • Next, in the PVLV_VS layer group find the VSPatchPosD1 layer (top one) and change the Fm_Input projection to Fm_RewTarg. Since we don't have any representation of time in this simple model, this projection provides a deterministic input to this component of the PVLV module so it can learn to anticipate the occurrence of phasic dopamine signals. Repeat this change for the VSPatchPosD2 layer. (Since we're not using aversive primary outcomes (e.g., tailshocks) in this model we don't have to bother with the complementary VSPatchNeg... layers as they won't affect anything.)

Running the Network

At last we're ready to actually run the network. Hit Init (Yes to Initialize the weights) and Run in the main ControlPanel (or in the LeabraTrain control panel which accesses the same methods). This will run the network. You should see the network learning within 10 epochs or so. Repeat a few times. (TIP: As before, you can speed things up a lot by watching the EpochOutputData graph view instead of watching the network itself running (in Network_1).

Once the network has learned, we can use the Step: | Trial:1 button on the ControlPanel to watch the network trial-by-trial. By default, the step goes one LeabraTrial at a time. You should be able to follow the flow of information through the PFC layers as follows:

  • On cue-time trials (A,B,C) you should see the PFCmnt_deep layers (behind/above PFCmnt) getting active, reflecting gating of information into maintenance ("working memory")
  • On probe-time trials (X,Y,Z) you should PFCout_deep layers get active for the 'X' trials, but not for the 'Y' or 'Z'. (It actually probably doesn't matter if output-gating occurs for the latter probe-types since the input alone is sufficient to get the correct answer. Repeat enough times to convince yourself that the gating of PFCout_deep always accompanies 'X' probe-time trials.
Figure 1: B-X sequence, probe X: quarter 1.
Figure 2: B-X sequence, probe X: quarter 2.
Figure 3: B-X sequence, probe X: quarter 3.

Now click on the the Step: | Quarter:1 button to watch the network quarter-by-quarter. You should be able to see the gating events occurring between Quarter:1 and Quarter:2, i.e., the PFC deep layers getting active. Try to understand the logic of the gating as you step quarter-by-quarter, noting the activity in the Matrix and, especially, the GPi layer.

The most interesting trial type is the 'X' probe-time trial, of course, since the right answer depends on the previous cue-time trial - this is precisely what motivated our adding a working memory capability here. Figures 1-3 at the right illustrates how the PBWM mechanism contributes to the correct answer for these trials, especially for the non-AX trial types (B-X, C-X) in which the most common Target response for the X must be overridden.

  • In Figure 1 you can see that after Quarter 1 of settling the X input through its weights to the Hidden layer favors a Target response as seen in the moderate activation (light red) in the 'T' unit (Target; O_T; right-most) of the Output layer. Note that the PFCout_deep layer (top right corner of the network) is not yet active at all (units are all gray).
  • In Figure 2, however, we see that by the end of Quarter 2 the gated activity in the PFCout_deep is now driving some competing 'N' unit (Non-Target; O_N; left-most) activity in Output, also via the Hidden layer where it is integrated with Input.
  • In Figure 3, we now see that the 'N' response has won the competition outright as a result of the maintained, then out-gated, PFCout_deep input. Note also how the the activity pattern in the Hidden layer evolves between Figure 1 to Figure 3, which is best thought of as representing either the Non-Target or Target response.

Thus, we can interpret the pattern of activity in the PFC layers that are driving the Non-Target response as being a representation of the 'B' cue from the prior trial. Conversely, for A-X trials a different pattern of activity in the PFC layers constitutes a representation of the prior 'A' cue (not shown).

Displaying Unit Names

It can sometimes make it easier to follow what is happening to change the display for the Input and/or Output layers (and sometimes other layers as well) to display the name of its most active unit, rather than just the raw unit activations. As a last little exercise we'll go ahead and do that.

There are two steps to this. First, we need to assign name labels to the individual units; and second, we'll need to set the network display to show the labels instead of the unit activations.

  • Select the Input layer (under layers not specs) under Network_1 at the bottom of the Navigator tree and click ON the SAVE_UNIT_NAMES flag near the top of its purple control panel in the middle Editor. Now fully open its sub-tree (e.g., double-click) and select unit names at the end. In the table that appears in Editor panel assign names for the units according to the unit geometry of the layer: A,B,C in the bottom row; X,Y,Z in the top row:


XYZ
ABC


  • Then do similar for the Output layer: i.e., N,T.
  • Now go to the Network_1 tab in the Visualizer and using the red arrow tool click on the Input layer's green-ish border until it turns red (indicating it has been selected). Then, context-click and select LayerView/Disp Output Name. Do the same for the Output layer.

Now Init and Run [and/or Step: | Trial:1 (or Quarter:1)] the network again and you'll see the label of the most active unit displayed as the network runs.

  • NOTE: The most active unit is defined by the end of the LeabraTrial, i.e., the end of the PLUS_PHASE. So, it is the Target values for the Output layer that is displayed each trial.

That is all we have for now. There is an even more complex version of the CPT-AX task (1-2 AXCPT) involving an outer-loop of 1 or 2 stimuli that determines what the inner-loop target sequence is (AX or BY). These same basic mechanisms can learn this more difficult task, though it takes longer. It is described in detail in the O'Reilly & Frank 2006 paper referenced above. You might notice a project called 12ax4s.proj in this same directory, but it has NOT yet been updated to the latest version of Emergent, nor of PBWM, so it will not work properly if at all.

Crystal Clear app display.png Using emergent