AXTut OutputData
From Emergent
(back to AX Tutorial)
Contents |
Monitoring, Analyzing, and Displaying Output Data
In this section we explore various ways of understanding better how the network is performing.
Graphing Learning Performance over Epochs
To see how your network is learning a given task, the first step is to generate a graph of the learning performance (sum squared error on the training patterns) over epochs. Fortunately, the default programs are already collecting this data for us, so we just need to display the graph.
The data is being recorded in the .data.gp.OutputData.EpochOutputData data table object in the data/OutputData subgroup. You should see a few rows of data from the previous run, and you should notice that the avg_sse column shows a general decrease in average sum-squared-error on the training patterns, ending in a 0, which is what stopped the training.
To graph this data, you just generate a graph view of this data table. As a general rule in the software, all view information is always generated by a given main object like a datatable or a network -- you don't create a graph and then associate data with it -- it goes the other way. The menu selection to create the graph view is View/New Graph View. This time, let's be adventurous and instead of putting this graph in a separate view frame, put it in the Network_0 frame.
If you happened to have put it in the wrong frame initially, don't worry -- just do the context menu over the frame view tab on the right (should be called EpochOutputData), and select Delete Frame. Note that there can be multiple views of the same underlying data.
You should see a graph appear in the upper right of your network display, showing a decreasing line from left to right. By default the line is (redundantly) color coded for the plot value. You can control this and many other features of the graph display in the graph control panel.
But wait, where is that panel? You should only see a .PanelTab.Network_0 tab in the middle (Editor) panel tabs. If you click on that guy, and look at the bottom, you'll see selectors for the different view objects within this one view frame (Network_0 and EpochOutputData Graph). Select the graph view tab at the bottom, and again mouse over the various controls and play around with them. As you can see, there are many different ways of configuring the graphs -- feel free to explore. Note that there are several other variables that you could plot, including average cycles to settle, and a count of the number of errors made across trials. Also see the wiki .Wiki.GraphView page for more details.
To see your graph updating in real-time, you can re-init and run the .programs.gp.LeabraAll_Std.LeabraTrain program: Init Run.
Arranging the 3d View
Although the .T3Tab.Network_0 view is sufficient, it could be configured to look better. We could shrink the graph view a bit, and orient it better. To do this (optional), click on the red arrow button to the right of the view, and then grab the upper horizontal bar of the small purple box in the lower-left hand corner of the graph view display. Drag this slowly down -- you'll see the green frame rotating as you do. Do this to the point where graph is angled more "head on". Similarly, you can grab the left vertical bar and rotate the graph to the left a bit to make it more face on. Next, grab any corner of the box, and shrink the view a bit (maybe to half or so of its original size). Finally, you can move the view down and to the right a bit, by clicking on the face of the cube (not on any of the purple elements), to fit in between the Hidden and Output layers.
When you've got it the way you want, you can press the eye button to resize the display to fit, and maybe Dolly zoom in a bit. You could perhaps pan to the right a bit with shift-mouse. When it looks good, hit the blueprint house button ("save home"), which saves this view configuration.
Recording Network Activations for a Grid View
Another common analysis task is to look at the pattern of activations across trials. To do this, we need to record activation values to our trial-level output data table (which was automatically created by the wizard). The easiest way to do this is to select the network object in the network view by clicking on the thin green frame surrounding the text display at the bottom of the network, and then using the right mouse button to get the context menu, and select [[.networks[0].MonitorVar()|MonitorVar]]. For the net_mon field, click and select the trial_netmon, which is for monitoring information at the trial level (the other one is for the epoch level). For the variable field, type in "act_m" (no quotes), to record minus (m) phase activations (act) (the monitor runs at the end of the trial, after the plus phase). This will record activations for all three layers in the network in one easy step (you could alternatively do MonitorVar on each of the layers individually, or on any other object in the network for that matter, and record any variable).
Next, we need to make a Grid View of the resulting data, which will be recorded in the .data.gp.OutputData.TrialOutputData object -- do a View/New Grid View, and again let's put this in the Network_0 frame. Follow the same general steps as before (see Arranging the 3d View above) to position this new grid view into the bottom right hand region of the view.
The grid view will not contain the new information until the .programs.gp.LeabraAll_Std.LeabraTrain program is Init and Run again. After doing that, you need to scroll the grid view display all the way over to the right -- there are too many columns to fit within the 5 columns that the standard grid view is configured to display. To do this, select the red arrow tool and drag the purple bar at the bottom of the TrialOutputData gridview all the way to the right. You should see some colored squares with the Input, Hidden, and Output column headers.
To really make things clean, you can select the column headers of the columns you don't want to display (e.g., ext_rew, minus cycles) and do context menu/View Properties and then hit the Hide button at the bottom of the dialog that comes up. Ideally, you'd just want to see the trial name, sse, and the different layer activation columns.
Also, because there are just 6 events, we can set the rows to display to 6 in the grid view control panel, to make the display fit just right.
Again, you can run your program and see it update the display.
Analyzing the Hidden Layer Representations
Now that we have some data from the network, we can perform some powerful analysis techniques on that data.
First, we can make a cluster plot of the Hidden Layer activations, to see if we can understand better what is being encoded. To do this, find the .data_proc.data_anal object under data_proc/data_anal in the left browser. This contains many useful analysis tools, organized by different topics in the buttons at the bottom. Select HighDim/Cluster, and set the following parameters (leave the rest at their defaults):
- view = on (generate a graph of the cluster data)
- src_data = TrialOutputData
- data_col_nm = Hidden_act_m (specifies the column of data that we want to cluster)
- name_col_name = trial_name (specifies the column with labels for the cluster nodes)
You should see a new graph show up, with the A,B,C,X,Y,Z labels grouped together into different clusters. Most likely, you should observe in a trained network that the A and X are grouped together, separate from the other items. Can you figure out why this would be the case?
Another way to process this high-dimensional activation pattern data is to project it down into 2 dimensions. Two techniques for this are principal components analysis (PCA) and multidimensiona scaling (MDS). To try PCA, select HighDim/PCA2dPrjn -- fill in the same info you did for the Cluster plot. You should see that the labels are distributed as points in a two-dimensional space, with the X-axis being the dimension (principal component) of the hidden layer activation patterns that captures the greatest amount of variance across patterns, and the Y-axis being the second strongest component. Accordingly, you should see A and X on the left or the right side of the graph, and the others on the other side. It is not clear what the vertical axis represents..
There are many more things one could do, but hopefully this gives a flavor. The next step: AXTut TaskProgram is to write a program to automatically generate our input patterns for training the network -- initially we'll start out with the simple task we ran already, but then we'll progressively expand to more complex tasks.
