Backprop 8.0 Update

From emergent
Revision as of 01:56, 31 August 2016 by Oreilly (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Back to Backpropagation or Changes_8.0.0

This page describes significant changes made to the Backpropagation implementation for version 8.0 of the software.

All of the supported options are now available in the basic BpUnitSpec and BpConSpec, with enumerated options for things like activation functions etc. This is necessary for supporting the GPU implementation which does not support C++-style class hierarchies very well, and is also much easier for exploring different options without having to create different spec types. Also, the use of function pointers to select the error function and decay type has been replaced with enumerated options, for similar reasons. Existing projects will convert those options automatically.

If you were using different types of BpUnitSpec's to support different activation functions, you need to replace those with a basic BpUnitSpec with the proper act_fun activation function selected.

The different momentum_type options available previously in BpConSpec have been removed, in favor of the AFTER_LRATE option, which is the standard form used in almost all models currently. This simplifies the code considerably.

Several major improvements were made as well, including:

  • There is now a single SOFTMAX act_fun option, which replaces the combination of ExpBpUnitSpec and SoftMaxBpUnitSpec from before -- it does the entire softmax computation internally, and uses the proper error derivative -- it is MUCH more stable and robust than the previous implementation.
  • The "dropout" regularizer has been added -- see this paper for full details:
  • There is a MAX_POOL act_fun option which computes the max over all of its recv connections, and only backpropagates error back to that max connection.
  • Connection weights can be shared across units, to support convolutional networks. This is supported in the TiledGpRFPrjnSpec ProjectionSpec which creates appropriate overlapping tiled connectivity patterns -- click the share_cons flag. With all of these options, it is now possible to support so-called Deep Convolutional Neural Networks, which have been shown to improve object recognition performance significantly (among other classification tasks).