Startup script
From Emergent
A startup script is a css script that controls what to do with a project when run without the gui (see nogui run).
NOTE: as of 4.0.7, it is preferable to create a startup program within the project itself, which then makes everything fully self-contained, and there are handy program elements that make it very easy to add custom startup arguments to set parameters in your project.
The Emergent system install directory (/usr/local/share/Emergent on linux and mac systems) contains a css_stdlib directory with various standard startup scripts for various algorithms.
Here is the leabra_startup.css script:
// startup and run a project in the background (nogui run or batch run), use like this:
// [nice +10] emergent -nogui -p my_project.proj -f leabra_startup.css [args...] &
// * nice +10 is optional but is nice to other users/processes if not running on
// a dedicated compute server
// * -nogui runs w/out the gui
// * -p my_project.proj is the name of the project to load
// * -f leabra_startup.css loads and runs this script
// * [args...] are other optional arguments as defined in the script, including:
// ** epochs=xxx number of epochs to run
// ** batches=xxx number of training batches to run
// ** tag=xxx additional text to append to log and network file names
// ** logdir=xxx subdirectory from current directory to put log files in
// * & at the end puts the job in the background.
void SetLogFile(String ext, String tag, String subdir, DataTable* dt, bool dmem=false) {
// if not sep log files by dmem, then only for first guy!
if(!dmem && (taMisc::dmem_proc != 0)) return;
// this function does all the work: gets file name from project
// and appends our specific stuff to it
String log_nm = dt->GetFileNameFmProject(ext, tag, subdir, dmem);
// tell the datatable object to save to this file name --
// last arg is whether to restrict to dmem proc 0..
dt->SaveDataLog(log_nm, false, !dmem);
// give the user some info:
if(taMisc::dmem_proc == 0)
cerr << "Saving data log of: " << dt->name << " to: " << log_nm << endl;
}
void Startup() {
if(.projects.size == 0) {
taMisc::Error("Error: project file not found!");
return;
}
taMisc::AddArgName("tag=", "FileTag");
taMisc::AddArgName("logdir=", "LogDir");
taMisc::AddArgName("epochs=", "EpochMax");
taMisc::AddArgName("batches=", "BatchMax");
taMisc::UpdateArgs();
if(taMisc::dmem_proc == 0)
cerr << "Loaded project: " << .projects[0].file_name << endl;
Program* train = .programs.LeabraTrain;
if(!train) {
taMisc::Error("Error: train program not found!");
return;
}
Program* batch = .programs.LeabraBatch;
if(!batch) {
taMisc::Error("Error: batch program not found!");
return;
}
if(taMisc::dmem_proc == 0)
cerr << "Initializing: " << batch->name << endl;
batch.Init(); // init first -- could have errors!
String tag = taMisc::FindArgByName("FileTag");
String logdir = taMisc::FindArgByName("LogDir");
train.SetVarFmArg("EpochMax", "max_epoch");
batch.SetVarFmArg("BatchMax", "max_batch");
// change last arg to true for debugging dmem..
SetLogFile(".epc.dat", tag, logdir, .data.EpochOutputData, false);
// trial-level data should always be dmem
SetLogFile(".trl.dat", tag, logdir, .data.TrialOutputData, true);
Program* saveweights = .programs.SaveWeights;
if(saveweights) { // propagate the tag..
saveweights.SetVar("tag", tag);
}
if(taMisc::dmem_proc == 0)
cerr << "Starting to run: " << batch->name << endl;
batch.Run();
}
Startup();
Contents |
Detailed Comments on the Script Elements
Although the code should be fairly readable as-is, here are a few comments to help explain things:
Specifying the list of possible args (options)
taMisc::AddArgName("tag=", "FileTag");
taMisc::AddArgName("logdir=", "LogDir");
taMisc::AddArgName("epochs=", "EpochMax");
taMisc::AddArgName("batches=", "BatchMax");
taMisc::UpdateArgs();
The software has an automatic argument list processor, and you can just add to the list of recognized arguments using the above functions. The first item in the AddArgName function is what the user types on the command line (with the value coming after the = sign (with no space), as in:
epochs=500
The second item is the unique "name" identifier for this argument, which you will use later to get the argument value. The two are not the same, because there can be multiple entries for a single underlying parameter if you want to give people flexibility in how they specify it. For example, you could add the following line:
taMisc::AddArgName("log_dir=", "LogDir");
to allow an alternate specification of the arg. The use of name=value formatting of the args is strongly encouraged, as it makes them position independent. If you just want a "flag" type of argument, do something like this:
taMisc::AddArgName("-my_flag", "MyFlag");
Reading the Arg Values
String tag = taMisc::FindArgByName("FileTag");
String logdir = taMisc::FindArgByName("LogDir");
This code shows the basic way of getting arg values. You specify the "name" of the argument to get the value (if any) passed by the user. If the user did not specify this argument, the resulting value is an empty string.
train.SetVarFmArg("EpochMax", "max_epoch");
batch.SetVarFmArg("BatchMax", "max_batch");
This second way of accessing arg values lets the program objects set a program variable (either in the program args or vars section) directly from an argument passed on startup, specified by the arg "name" as above. So, the program train has a variable called max_epoch which is set by the startup argument named "EpochMax", if it was passed (if it wasn't passed by the user, then nothing happens -- these args are optional).
This SetVarFmArg function will also print out a message saying that it set the parameter (and what it set it to) if it did, or what the existing value of the variable is if it didn't set the variable.
For flag-style arguments, you can do something like this:
if(taMisc::CheckArgByName("MyFlag")) {
cerr << "my flag was set!!" << endl;
// do whatever is implied by this flag being set!!
}
Accessing Program Objects
Program* train = .programs.LeabraTrain;
if(!train) {
taMisc::Error("Error: train program not found!");
return;
}
This code tries to find a given program named LeabraTrain, and sets a local variable to refer to it. If this program is not found, then it prints out an error message and exits. The local program variable is then used for setting arg values (as above) or to initialize and run the program:
batch.Init(); // init first -- could have errors! ... batch.Run();
Saving Log Files
If you don't setup the system to record the results of running, there isn't much point to it all. The script contains a little subroutine for saving the data that is stored in the OutputData datatables (the EpochOutputData and the TrialOutputData):
void SetLogFile(String ext, String tag, String subdir, DataTable* dt, bool dmem=false) {
// if not sep log files by dmem, then only for first guy!
if(!dmem && (taMisc::dmem_proc != 0)) return;
// this function does all the work: gets file name from project
// and appends our specific stuff to it
String log_nm = dt->GetFileNameFmProject(ext, tag, subdir, dmem);
// tell the datatable object to save to this file name --
// last arg is whether to restrict to dmem proc 0..
dt->SaveDataLog(log_nm, false, !dmem);
// give the user some info:
if(taMisc::dmem_proc == 0)
cerr << "Saving data log of: " << dt->name << " to: " << log_nm << endl;
}
This function is then called in the main code for each of the datatables:
// change last arg to true for debugging dmem..
SetLogFile(".epc.dat", tag, logdir, .data.EpochOutputData, false);
// trial-level data should always be dmem
SetLogFile(".trl.dat", tag, logdir, .data.TrialOutputData, true);
The stuff about dmem refers to distributed memory parallel processing, as on a multi-node cluster, using the mpi protocol. If you're not doing this, then just ignore anything having to do with dmem because it doesn't make any difference. If you are doing it, then you must know what you're doing! (or read more at the dmem link).
Saving Networks/Weights
The other main thing you'd want to save when you run your project is the weights (or entire network object) after training. This is standardly performed by the SaveWeights program (available in the program library), which can be called at the end of training, or during training to checkpoint, or whenever you want. Just insert a ProgramCall program element in the appropriate location in your programs in the project. The startup script assumes that this has all been configured before running, and so it does not do anything special about saving weights, except this one bit:
Program* saveweights = .programs.SaveWeights;
if(saveweights) { // propagate the tag..
saveweights.SetVar("tag", tag);
}
which propagates the user-supplied "tag" to the SaveWeights program.
