CCNBook/Networks/kWTA Equations

Back to Networks Chapter

This sub-topic describes the k-winners-take-all (kWTA) form of inhibition, which was the default in the 1st Edition of this wiki textbook, having now been replaced with the FFFB form of inhibition described in the main chapter. Figure 3.1.1: How kWTA is computed as function of distribution of excitation of neurons in a layer. Inhibitory conductance g_i is calculated as point between k and k+1 most excited neurons, in terms of the amount of inhibition required to keep the membrane potential Vm right at firing threshold θ (g_i^θ). Thus, exactly k neurons will be above their firing threshold and active, while the remainder will be below threshold and inactive. The shape of the distribution determines how strongly the active units are activated, as a function of the gap between their excitation level and the g_i inhibition -- when these are very close (middle case), activation levels are very low, and when they are very far apart (right case) activation levels are high. But always exactly k units are active. Figure 3.1.2: How average-based kWTA is computed as function of distribution of excitation of neurons in a layer. Inhibitory conductance g_i is calculated as a point (determined by proportion pt) between the average over the top k vs. average of the remainder (avg shown as dashed lines). Unlike the exact simple kWTA computation, this average-based version allows for more or less activity depending on the shape of the distribution, as shown in the middle and right cases.

Although the combination of feedforward and feedback inhibition dynamics in inhibitory interneurons is capable of producing robust activity regulation, it only does so when the parameters are set correctly, within a fairly narrow range. Changes due to learning or other modulations of excitability of neurons require adapting the parameters. (In the brain, this is likely accomplished with the help of glia in addition to other homeostatic mechanisms that enable the system to adapt the strength of inhibition over time to keep things in the right range.) Also, even when the parameters are set correctly, the feedback nature of inhibition requires that the time steps for computing the neuron update equations be very small -- otherwise numerical instability can result (e.g., wild oscillations).

For these reasons, we developed a simple set of approximations to the net effects of the inhibitory interneurons, known as the k-winners-take-all (kWTA) inhibitory functions. These equations directly compute a level of inhibitory conductance (g_i) for all the neurons in a layer, with this value placed so as to keep k units above their firing threshold (where k is a parameter set by the modeler, e.g., 15%), with the remainder below threshold. Thus, these equations capture the set point nature of inhibitory feedback, analogous to setting a desired temperature for the AC unit -- the kWTA function ensures that the "temperature" doesn't get too much above k.

To see how the basic form of kWTA is computed, see Figure 3.1.1Figure 3.1.1. First, all the neurons in the layer are sorted (rank ordered) in terms of the level of excitatory input they are receiving at the time. Counting down from the most excited neuron, we place a dividing line between the k and k+1 unit in this rank-ordered list. We then compute an amount of inhibitory conductance g_i that will inhibit the k+1 and beyond neurons to the point that they are below threshold, while not being too high to inhibit the top k neurons. As we shown in kWTA Equations, this can be done very straightforwardly by using the equilibrium membrane potential equation from the Neuron Chapter. We can compute the amount of inhibition that will place each neuron right at its firing threshold, denoted $g_i^\Theta$ -- this is monotonically related to the excitatory input into the neuron, and is the value that is actually plotted in the figure. Thus, the actual inhibition level can be read off directly from this curve at the k and k+1 points, as shown in the figure -- everyone above k will have an inhibition value that is less than what it would take to keep them at threshold, so they get over threshold, while the remainder have an inhibition value that is greater than their threshold inhibition level, and thus are inhibited below threshold.

The average-based version of the kWTA function introduces a bit more flexibility in the overall activity levels within a layer, by computing the inhibitory conductance for the layer in terms of the average inhibitory threshold values for the top k vs. remainder neurons. As shown in Figure 3.1.2Figure 3.1.2, this averaging process enables the actual g_i inhibitory value to sometimes be higher or lower than the point between the k and k+1 neuron, depending on the overall shape of the distribution of excitation within the layer.

Typically, we use the average-based kWTA for hidden layers, where the extra flexibility is very useful in enabling the layer to develop appropriate representations for different inputs, which may require different numbers of overall active neurons. For output layers with a prescribed number of active neurons, the basic kWTA works well to ensure that this number of neurons are always active.

Finally, we emphasize that although the actual computational mechanism for these kWTA functions is biologically implausible, they nevertheless provide a good approximation for the behavior of networks governed by inhibitory interneuron inhibition. Thus, the net result is the same, and the computational advantages of kWTA are critical for enabling models to be run quickly and efficiently. As with the use of the rate code approximation instead of discrete spiking, the best way to mitigate against the potential limitations of using the kWTA approximation is to compare models using each mechanism, and determine what if any systematic differences exist. More of this kind of exploration needs to be done, now that computer hardware is significantly faster and such models can be relatively easily run.

kWTA Equations

The amount of inhibition to keep a neuron right at its firing threshold can be computed from the equilibrium membrane potential equation from the Neuron chapter, and results in:

• $g_i^{\Theta} = \frac{g^*_e \bar{g_e} (E_e - \Theta) + g_l \bar{g_l} (E_l - \Theta)}{\Theta - E_i}$

The basic kWTA function places the actual inhibition between the k and k+1 unit, rank ordered according to their threshold inhibition levels:

• $g_i = g_i^{\Theta}(k+1) + q (g_i^{\Theta}(k) - g_i^{\Theta}(k+1))$

Where q is the kwta_pt value in the simulator that determines where exactly to place the inhibition between the k and k+1 neurons.

Average Based

The average-based kWTA function computes averages as follows. For the top k average, it is:

• $\langle g_i^{\Theta} \rangle_k = \frac{1}{k} \sum_{i=1}^k g_i^{\Theta}(i)$

And for the average of the remainder of n-k units (where n is total number in the layer), is:

• $\langle g_i^{\Theta} \rangle_{n-k} = \frac{1}{n-k} \sum_{i=k+1}^n g_i^{\Theta}(i)$

The same way of placing the inhibition a fixed proportional distance between these two averages is used:

• $g_i = \langle g_i^{\Theta} \rangle_{n-k} + q (\langle g_i^{\Theta} \rangle_k - \langle g_i^{\Theta} \rangle_{n-k})$