CCNBook/Networks/Energy and Harmony
Back to Networks Chapter
This sub-section describes the Hopfield energy and Smolensky Harmony equations, and how they help us understand more formally what our networks are doing as they settle into an attractor state.
The Hopfield energy equation is:
where x and y represent the sending and receiving unit activations (indexed by i and j), respectively, and w is the weight between them.
Harmony is literally the same thing without the minus sign:
You can see that Harmony is maximized to the extent that, for each pair of sending and receiving units, the activations of these units x_i and y_j are consistent with the weight between these two units. If the weight is large and positive, the network is configured such that it is harmonious if these two units are both active together. If the weight is negative (a simple version of inhibitory projections), then those units contribute to greater harmony only if they have opposite sign (one is active and the other not active).
A key feature of these equations is that local updates drive reliable global effects on energy or Harmony (decreasing the energy or increasing Harmony). To see this, we can use the mathematics of calculus to take the derivative of the global equation with respect to changes in the receiving unit's activation:
Taking the derivative allows us to find the maximum of a function, which occurs when the derivative is zero. So, this gives us a prescriptive formula for deciding how y_j should be changed (updated) as a function of inputs and weights so as to maximize Harmony. You might recognize this equation as essentially the net excitatory conductance or net input to a neuron, from the Neuron Chapter. This means that updating units with a linear activation function (where activation y = net input directly) would serve to maximize Harmony or minimize energy. To accommodate a non-linear activation function (e.g., a "sigmoidal" function of the same general shape as the XX1 function), one needs to introduce and additional "penalty" term (called entropy in the Hopfield framework, and stress in the Smolensky one), that essentially drives the saturation of the neural activation function for high or low values of net input.